{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "accredited-radical",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 5.19 s (started: 2021-07-25 18:19:14 +08:00)\n"
     ]
    }
   ],
   "source": [
    "# 自动计算cell的计算时间\n",
    "%load_ext autotime\n",
    "\n",
    "#设置使用的gpu\n",
    "import tensorflow as tf\n",
    "\n",
    "gpus = tf.config.list_physical_devices(\"GPU\")\n",
    "\n",
    "if gpus:\n",
    "   \n",
    "    gpu0 = gpus[2] #如果有多个GPU，仅使用第0个GPU\n",
    "    tf.config.experimental.set_memory_growth(gpu0, True) #设置GPU显存用量按需使用\n",
    "    # 或者也可以设置GPU显存为固定使用量(例如：4G)\n",
    "    #tf.config.experimental.set_virtual_device_configuration(gpu0,\n",
    "    #    [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=4096)]) \n",
    "    tf.config.set_visible_devices([gpu0],\"GPU\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "respective-tuner",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 14.1 ms (started: 2021-07-25 18:19:19 +08:00)\n"
     ]
    }
   ],
   "source": [
    "%config InlineBackend.figure_format='svg' #矢量图设置，让绘图更清晰"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "average-fundamental",
   "metadata": {},
   "source": [
    "# 机器学习基础"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "strategic-software",
   "metadata": {},
   "source": [
    "本章包含以下内容：\n",
    "* 理解泛化和优化之间的张力，这是机器学习的基本问题\n",
    "* 机器学习模型的评估方法\n",
    "* 改进模型拟合的最佳实践\n",
    "* 实现更好泛化的最佳实践"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hydraulic-threat",
   "metadata": {},
   "source": [
    "在第 4 章中的三个实际示例之后，您应该开始熟悉如何使用神经网络处理分类和回归问题，并且您已经目睹了机器学习的核心问题：过度拟合。 本章将把你对机器学习的一些新直觉形式化为一个坚实的概念框架，强调准确模型评估的重要性以及训练和泛化之间的平衡。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "liberal-hypothesis",
   "metadata": {},
   "source": [
    "## 泛化：机器学习的目标"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "forbidden-glass",
   "metadata": {},
   "source": [
    "在第 4 章介绍的三个示例中——预测电影评论、主题分类和房价回归——我们将数据分成训练集、验证集和测试集。 不使用它们训练的相同数据评估模型的原因很快就变得明显了：仅仅几个 epoch 之后，前所未见的数据上的性能就开始与训练数据上的性能有所不同——随着训练的进行，这种性能总是有所提高。 模型开始过度拟合。 过拟合发生在每个机器学习问题中。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pregnant-carnival",
   "metadata": {},
   "source": [
    "机器学习的基本问题是优化和泛化之间的张力。 优化是指调整模型以在训练数据上获得最佳性能的过程（学习机器学习），而泛化是指经过训练的模型在它以前从未见过的数据上的表现。 游戏的目标当然是获得良好的泛化，但你无法控制泛化； 您只能将模型拟合到其训练数据中。 如果你这样做，过度拟合就会开始，泛化就会受到影响。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sweet-morgan",
   "metadata": {},
   "source": [
    "但是是什么导致过拟合呢？ 如何实现良好的泛化？"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "architectural-expansion",
   "metadata": {},
   "source": [
    "### 欠拟合和过拟合"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fundamental-dealer",
   "metadata": {},
   "source": [
    "对于您在前一章中看到的所有模型，保留验证数据的性能开始随着训练的进行而提高，然后不可避免地在一段时间后达到顶峰。 这种模式（如图 5.1 所示）是通用的。 您将在任何模型类型和任何数据集上看到它。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ultimate-repository",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gst75sd84mj31ey0qgmyu.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "boolean-organization",
   "metadata": {},
   "source": [
    "在训练开始时，优化和泛化是相关的：训练数据的损失越低，测试数据的损失就越低。 在发生这种情况时，您的模型被认为是欠拟合的：仍有待取得进展； 该网络尚未对训练数据中的所有相关模式进行建模。 但是在对训练数据进行一定次数的迭代后，泛化停止提高，验证指标停滞，然后开始降级：模型开始过度拟合。 也就是说，它开始学习特定于训练数据但在涉及新数据时具有误导性或无关性的模式。\n",
    "\n",
    "当您的数据嘈杂、涉及不确定性或包含稀有特征时，特别有可能发生过度拟合。 我们来看具体的例子。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pursuant-printer",
   "metadata": {},
   "source": [
    "**嘈杂的训练数据**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "intended-absorption",
   "metadata": {},
   "source": [
    "在现实世界的数据集中，某些输入无效是很常见的。 例如，MNIST 数字可能是全黑图像。 或者像这样："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "deluxe-bradford",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gstcwq1qyuj30vs0n2n05.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "agricultural-prospect",
   "metadata": {},
   "source": [
    "这些是什么？ 我也不知道。 但它们都是 MNIST 训练集的一部分。 然而，更糟糕的是，完全有效的输入最终被错误标记。 像这些："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tight-annex",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gstcxooaxoj31ay0dg76d.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "solar-topic",
   "metadata": {},
   "source": [
    "如果模型不顾一切地合并这些异常值，它的泛化性能就会下降，如图 5.4 所示。 例如，看起来非常接近上面错误标记的 5 的 5 可能最终被归类为 7。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "temporal-pension",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gstcyrdcjlj31b80lkacc.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "coral-algebra",
   "metadata": {},
   "source": [
    "**模棱两可的特征**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "racial-marsh",
   "metadata": {},
   "source": [
    "并非所有数据噪声都来自不准确——即使是完全干净且标记整齐的数据，当问题涉及不确定性和歧义时，也可能存在噪声。 在分类任务中，经常出现输入特征空间的某些区域同时与多个类相关联的情况。 假设您正在开发一个模型，该模型获取香蕉的图像并预测香蕉是未成熟、成熟还是腐烂。 这些类别没有客观界限，因此相同的图片可能被不同的人工标记者归类为未成熟或成熟的。 同样，许多问题都涉及随机性。 您可以使用大气压力数据来预测明天是否会下雨，但完全相同的测量值之后可能有时会下雨，有时会出现晴朗的天空——有一定的概率。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "constitutional-lotus",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gstd0kblawj31ck0h4aef.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "treated-machinery",
   "metadata": {},
   "source": [
    "由于对特征空间的模糊区域过于自信，模型可能会过度拟合此类概率数据，如图 5.5 所示。 更稳健的拟合将忽略单个数据点并着眼于更大的图景。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hybrid-karma",
   "metadata": {},
   "source": [
    "**罕见的特征和虚假的相关性**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "secure-emperor",
   "metadata": {},
   "source": [
    "如果您一生中只见过两只橙色虎斑猫，而且它们都非常反社会，那么您可能会推断橙色虎斑猫通常很可能是反社会的。 这是过度拟合：如果你接触过更多种类的猫，包括更多的橙色猫，你就会知道猫的颜色与性格没有很好的相关性。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "finite-asian",
   "metadata": {},
   "source": [
    "同样，在包含稀有特征值的数据集上训练的机器学习模型极易过拟合。 在情感分类任务中，如果“cherimoya”（一种原产于安第斯山脉的水果）一词仅出现在训练数据中的一个文本中，而该文本恰好是负面的，那么正则化不良的模型可能会给出非常高的评价 重视这个词，并始终将提及 cherimoyas 的新文本归类为负面。 然而，客观地说，樱桃没有任何负面影响。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "successful-rolling",
   "metadata": {},
   "source": [
    "重要的是，一个特征值不需要只出现几次就可以导致虚假的相关性。 考虑一个出现在训练数据 100 个样本中的词，它与正面情绪相关的概率为 54%，负面情绪概率为 46%。 这种差异很可能是一个完整的统计侥幸，但是，您的模型很可能会学习利用该特征进行分类任务。 这是过拟合的最常见来源之一。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "warming-certificate",
   "metadata": {},
   "source": [
    "这是一个引人注目的例子。 以 MNIST 为例。 通过将 784 个白噪声维度连接到现有的 784 个数据维度来创建一个新的训练集——所以现在有一半的数据是噪声。 为了进行比较，还可以通过串联 784 个全零维度来创建等效数据集。 我们对无意义特征的串联根本不会影响数据的信息内容：我们只是添加了一些东西。 人类分类的准确性根本不会受到这些转换的影响。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "frank-backing",
   "metadata": {},
   "source": [
    "> 清单 5.1 向 MNIST 添加白噪声通道或全零通道"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "passive-domestic",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.91 s (started: 2021-07-25 20:51:01 +08:00)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.datasets import mnist\n",
    "import numpy as np\n",
    "\n",
    "(train_images, train_labels), _ = mnist.load_data()\n",
    "train_images = train_images.reshape((60000, 28 * 28))\n",
    "train_images = train_images.astype(\"float32\") / 255\n",
    "\n",
    "train_images_with_noise_channels = np.concatenate(\n",
    "    [train_images, np.random.random((len(train_images), 784))], \n",
    "    axis=1)\n",
    "train_images_with_zeros_channels = np.concatenate(\n",
    "    [train_images, np.zeros((len(train_images), 784))], \n",
    "    axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "understanding-committee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 1568)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 11.1 ms (started: 2021-07-25 18:20:09 +08:00)\n"
     ]
    }
   ],
   "source": [
    "train_images_with_noise_channels.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sudden-economics",
   "metadata": {},
   "source": [
    "现在，让我们在这两个训练集上训练第 2 章中的模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "activated-lobby",
   "metadata": {},
   "source": [
    "> 清单 5.2 使用噪声通道或全零通道在 MNIST 数据上训练相同模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "simple-brighton",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 798 µs (started: 2021-07-25 18:31:26 +08:00)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "def get_model():\n",
    "    model = keras.Sequential([\n",
    "        layers.Dense(512, activation=\"relu\"),\n",
    "        layers.Dense(10, activation=\"softmax\")\n",
    "    ])\n",
    "    model.compile(optimizer=\"rmsprop\",\n",
    "                  loss=\"sparse_categorical_crossentropy\",\n",
    "                  metrics=[\"accuracy\"])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "arranged-situation",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "375/375 [==============================] - 4s 8ms/step - loss: 0.6307 - accuracy: 0.8129 - val_loss: 0.2998 - val_accuracy: 0.9117\n",
      "Epoch 2/10\n",
      "375/375 [==============================] - 4s 10ms/step - loss: 0.2488 - accuracy: 0.9223 - val_loss: 0.2181 - val_accuracy: 0.9330\n",
      "Epoch 3/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.1590 - accuracy: 0.9509 - val_loss: 0.1457 - val_accuracy: 0.9569\n",
      "Epoch 4/10\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 0.1107 - accuracy: 0.9651 - val_loss: 0.1354 - val_accuracy: 0.9584\n",
      "Epoch 5/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0819 - accuracy: 0.9743 - val_loss: 0.1286 - val_accuracy: 0.9612\n",
      "Epoch 6/10\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0590 - accuracy: 0.9812 - val_loss: 0.1214 - val_accuracy: 0.9670\n",
      "Epoch 7/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0448 - accuracy: 0.9856 - val_loss: 0.1093 - val_accuracy: 0.9695\n",
      "Epoch 8/10\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0325 - accuracy: 0.9898 - val_loss: 0.1498 - val_accuracy: 0.9604\n",
      "Epoch 9/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0258 - accuracy: 0.9924 - val_loss: 0.1296 - val_accuracy: 0.9684\n",
      "Epoch 10/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0194 - accuracy: 0.9942 - val_loss: 0.1414 - val_accuracy: 0.9673\n",
      "time: 31.9 s (started: 2021-07-25 18:31:32 +08:00)\n"
     ]
    }
   ],
   "source": [
    "model = get_model()\n",
    "\n",
    "history_noise = model.fit(\n",
    "    train_images_with_noise_channels, \n",
    "    train_labels,\n",
    "    epochs=10,\n",
    "    batch_size=128,\n",
    "    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "whole-choice",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "375/375 [==============================] - 4s 10ms/step - loss: 0.2912 - accuracy: 0.9151 - val_loss: 0.1554 - val_accuracy: 0.9557\n",
      "Epoch 2/10\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 0.1202 - accuracy: 0.9646 - val_loss: 0.1064 - val_accuracy: 0.9682\n",
      "Epoch 3/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0783 - accuracy: 0.9770 - val_loss: 0.0892 - val_accuracy: 0.9733\n",
      "Epoch 4/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0566 - accuracy: 0.9831 - val_loss: 0.0845 - val_accuracy: 0.9764\n",
      "Epoch 5/10\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0419 - accuracy: 0.9874 - val_loss: 0.0851 - val_accuracy: 0.9749\n",
      "Epoch 6/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0320 - accuracy: 0.9904 - val_loss: 0.0844 - val_accuracy: 0.9777\n",
      "Epoch 7/10\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 0.0247 - accuracy: 0.9924 - val_loss: 0.0818 - val_accuracy: 0.9768\n",
      "Epoch 8/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0188 - accuracy: 0.9946 - val_loss: 0.0823 - val_accuracy: 0.9792\n",
      "Epoch 9/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0137 - accuracy: 0.9964 - val_loss: 0.0871 - val_accuracy: 0.9797\n",
      "Epoch 10/10\n",
      "375/375 [==============================] - 4s 10ms/step - loss: 0.0113 - accuracy: 0.9970 - val_loss: 0.0923 - val_accuracy: 0.9778\n",
      "time: 33.2 s (started: 2021-07-25 18:32:07 +08:00)\n"
     ]
    }
   ],
   "source": [
    "model = get_model()\n",
    "\n",
    "history_zeros = model.fit(\n",
    "    train_images_with_zeros_channels, \n",
    "    train_labels,\n",
    "    epochs=10,\n",
    "    batch_size=128,\n",
    "    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "isolated-trustee",
   "metadata": {},
   "source": [
    "> 清单 5.3 绘制验证准确性比较"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "immune-grenada",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f84481153a0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg height=\"277.314375pt\" version=\"1.1\" viewBox=\"0 0 392.14375 277.314375\" width=\"392.14375pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2021-07-25T18:34:34.757018</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.4.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 277.314375 \n",
       "L 392.14375 277.314375 \n",
       "L 392.14375 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 50.14375 239.758125 \n",
       "L 384.94375 239.758125 \n",
       "L 384.94375 22.318125 \n",
       "L 50.14375 22.318125 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"mf5687a3b0c\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"99.180114\" xlink:href=\"#mf5687a3b0c\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(95.998864 254.356562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"166.816477\" xlink:href=\"#mf5687a3b0c\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(163.635227 254.356562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"234.452841\" xlink:href=\"#mf5687a3b0c\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 6 -->\n",
       "      <g transform=\"translate(231.271591 254.356562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" id=\"DejaVuSans-36\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"302.089205\" xlink:href=\"#mf5687a3b0c\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 8 -->\n",
       "      <g transform=\"translate(298.907955 254.356562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" id=\"DejaVuSans-38\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-38\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"369.725568\" xlink:href=\"#mf5687a3b0c\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(363.363068 254.356562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\n",
       "        <path d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- Epochs -->\n",
       "     <g transform=\"translate(199.628125 268.034687)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 628 4666 \n",
       "L 3578 4666 \n",
       "L 3578 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2753 \n",
       "L 3481 2753 \n",
       "L 3481 2222 \n",
       "L 1259 2222 \n",
       "L 1259 531 \n",
       "L 3634 531 \n",
       "L 3634 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" id=\"DejaVuSans-45\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" id=\"DejaVuSans-70\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" id=\"DejaVuSans-63\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" id=\"DejaVuSans-68\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" id=\"DejaVuSans-73\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-45\"/>\n",
       "      <use x=\"63.183594\" xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use x=\"126.660156\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"187.841797\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"242.822266\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"306.201172\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"mf0a2481da1\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#mf0a2481da1\" y=\"234.719483\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0.91 -->\n",
       "      <g transform=\"translate(20.878125 238.518702)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" id=\"DejaVuSans-2e\" transform=\"scale(0.015625)\"/>\n",
       "        <path d=\"M 703 97 \n",
       "L 703 672 \n",
       "Q 941 559 1184 500 \n",
       "Q 1428 441 1663 441 \n",
       "Q 2288 441 2617 861 \n",
       "Q 2947 1281 2994 2138 \n",
       "Q 2813 1869 2534 1725 \n",
       "Q 2256 1581 1919 1581 \n",
       "Q 1219 1581 811 2004 \n",
       "Q 403 2428 403 3163 \n",
       "Q 403 3881 828 4315 \n",
       "Q 1253 4750 1959 4750 \n",
       "Q 2769 4750 3195 4129 \n",
       "Q 3622 3509 3622 2328 \n",
       "Q 3622 1225 3098 567 \n",
       "Q 2575 -91 1691 -91 \n",
       "Q 1453 -91 1209 -44 \n",
       "Q 966 3 703 97 \n",
       "z\n",
       "M 1959 2075 \n",
       "Q 2384 2075 2632 2365 \n",
       "Q 2881 2656 2881 3163 \n",
       "Q 2881 3666 2632 3958 \n",
       "Q 2384 4250 1959 4250 \n",
       "Q 1534 4250 1286 3958 \n",
       "Q 1038 3666 1038 3163 \n",
       "Q 1038 2656 1286 2365 \n",
       "Q 1534 2075 1959 2075 \n",
       "z\n",
       "\" id=\"DejaVuSans-39\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-39\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-31\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#mf0a2481da1\" y=\"205.649947\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.92 -->\n",
       "      <g transform=\"translate(20.878125 209.449165)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-39\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#mf0a2481da1\" y=\"176.58041\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.93 -->\n",
       "      <g transform=\"translate(20.878125 180.379629)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" id=\"DejaVuSans-33\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-39\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#mf0a2481da1\" y=\"147.510874\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.94 -->\n",
       "      <g transform=\"translate(20.878125 151.310093)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-39\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#mf0a2481da1\" y=\"118.441338\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.95 -->\n",
       "      <g transform=\"translate(20.878125 122.240557)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" id=\"DejaVuSans-35\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-39\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#mf0a2481da1\" y=\"89.371802\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0.96 -->\n",
       "      <g transform=\"translate(20.878125 93.17102)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-39\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#mf0a2481da1\" y=\"60.302265\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 0.97 -->\n",
       "      <g transform=\"translate(20.878125 64.101484)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 525 4666 \n",
       "L 3525 4666 \n",
       "L 3525 4397 \n",
       "L 1831 0 \n",
       "L 1172 0 \n",
       "L 2766 4134 \n",
       "L 525 4134 \n",
       "L 525 4666 \n",
       "z\n",
       "\" id=\"DejaVuSans-37\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-39\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-37\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#mf0a2481da1\" y=\"31.232729\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 0.98 -->\n",
       "      <g transform=\"translate(20.878125 35.031948)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-39\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-38\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_15\">\n",
       "     <!-- Accuracy -->\n",
       "     <g transform=\"translate(14.798438 153.86625)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 2188 4044 \n",
       "L 1331 1722 \n",
       "L 3047 1722 \n",
       "L 2188 4044 \n",
       "z\n",
       "M 1831 4666 \n",
       "L 2547 4666 \n",
       "L 4325 0 \n",
       "L 3669 0 \n",
       "L 3244 1197 \n",
       "L 1141 1197 \n",
       "L 716 0 \n",
       "L 50 0 \n",
       "L 1831 4666 \n",
       "z\n",
       "\" id=\"DejaVuSans-41\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" id=\"DejaVuSans-75\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" id=\"DejaVuSans-72\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" id=\"DejaVuSans-79\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-41\"/>\n",
       "      <use x=\"66.658203\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"121.638672\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"176.619141\" xlink:href=\"#DejaVuSans-75\"/>\n",
       "      <use x=\"239.998047\" xlink:href=\"#DejaVuSans-72\"/>\n",
       "      <use x=\"281.111328\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"342.390625\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"397.371094\" xlink:href=\"#DejaVuSans-79\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_14\">\n",
       "    <path clip-path=\"url(#p33b56e0cfd)\" d=\"M 65.361932 229.874489 \n",
       "L 99.180114 167.859468 \n",
       "L 132.998295 98.334841 \n",
       "L 166.816477 93.974554 \n",
       "L 200.634659 85.980317 \n",
       "L 234.452841 69.023104 \n",
       "L 268.271023 61.755727 \n",
       "L 302.089205 88.160548 \n",
       "L 335.907386 64.904872 \n",
       "L 369.725568 68.05419 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_15\">\n",
       "    <path clip-path=\"url(#p33b56e0cfd)\" d=\"M 65.361932 101.726388 \n",
       "L 99.180114 65.389502 \n",
       "L 132.998295 50.612346 \n",
       "L 166.816477 41.649369 \n",
       "L 200.634659 46.00983 \n",
       "L 234.452841 38.015594 \n",
       "L 268.271023 40.438053 \n",
       "L 302.089205 33.655133 \n",
       "L 335.907386 32.201761 \n",
       "L 369.725568 37.773365 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 50.14375 239.758125 \n",
       "L 50.14375 22.318125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 384.94375 239.758125 \n",
       "L 384.94375 22.318125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 50.14375 239.758125 \n",
       "L 384.94375 239.758125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 50.14375 22.318125 \n",
       "L 384.94375 22.318125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"text_16\">\n",
       "    <!-- Effect of noise channels on validation accuracy -->\n",
       "    <g transform=\"translate(76.04875 16.318125)scale(0.12 -0.12)\">\n",
       "     <defs>\n",
       "      <path d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" id=\"DejaVuSans-66\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" id=\"DejaVuSans-65\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" id=\"DejaVuSans-74\" transform=\"scale(0.015625)\"/>\n",
       "      <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" id=\"DejaVuSans-6e\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" id=\"DejaVuSans-69\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" id=\"DejaVuSans-6c\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 563 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2284 0 \n",
       "L 1503 0 \n",
       "L 191 3500 \n",
       "z\n",
       "\" id=\"DejaVuSans-76\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" id=\"DejaVuSans-64\" transform=\"scale(0.015625)\"/>\n",
       "     </defs>\n",
       "     <use xlink:href=\"#DejaVuSans-45\"/>\n",
       "     <use x=\"63.183594\" xlink:href=\"#DejaVuSans-66\"/>\n",
       "     <use x=\"98.388672\" xlink:href=\"#DejaVuSans-66\"/>\n",
       "     <use x=\"133.59375\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "     <use x=\"195.117188\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "     <use x=\"250.097656\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "     <use x=\"289.306641\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"321.09375\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "     <use x=\"382.275391\" xlink:href=\"#DejaVuSans-66\"/>\n",
       "     <use x=\"417.480469\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"449.267578\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "     <use x=\"512.646484\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "     <use x=\"573.828125\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"601.611328\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     <use x=\"653.710938\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "     <use x=\"715.234375\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"747.021484\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "     <use x=\"802.001953\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "     <use x=\"865.380859\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "     <use x=\"926.660156\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "     <use x=\"990.039062\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "     <use x=\"1053.417969\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "     <use x=\"1114.941406\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "     <use x=\"1142.724609\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     <use x=\"1194.824219\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"1226.611328\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "     <use x=\"1287.792969\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "     <use x=\"1351.171875\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"1382.958984\" xlink:href=\"#DejaVuSans-76\"/>\n",
       "     <use x=\"1442.138672\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "     <use x=\"1503.417969\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "     <use x=\"1531.201172\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"1558.984375\" xlink:href=\"#DejaVuSans-64\"/>\n",
       "     <use x=\"1622.460938\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "     <use x=\"1683.740234\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "     <use x=\"1722.949219\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"1750.732422\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "     <use x=\"1811.914062\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "     <use x=\"1875.292969\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"1907.080078\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "     <use x=\"1968.359375\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "     <use x=\"2023.339844\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "     <use x=\"2078.320312\" xlink:href=\"#DejaVuSans-75\"/>\n",
       "     <use x=\"2141.699219\" xlink:href=\"#DejaVuSans-72\"/>\n",
       "     <use x=\"2182.8125\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "     <use x=\"2244.091797\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "     <use x=\"2299.072266\" xlink:href=\"#DejaVuSans-79\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 146.121875 234.758125 \n",
       "L 377.94375 234.758125 \n",
       "Q 379.94375 234.758125 379.94375 232.758125 \n",
       "L 379.94375 204.401875 \n",
       "Q 379.94375 202.401875 377.94375 202.401875 \n",
       "L 146.121875 202.401875 \n",
       "Q 144.121875 202.401875 144.121875 204.401875 \n",
       "L 144.121875 232.758125 \n",
       "Q 144.121875 234.758125 146.121875 234.758125 \n",
       "z\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_16\">\n",
       "     <path d=\"M 148.121875 210.500312 \n",
       "L 168.121875 210.500312 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_17\"/>\n",
       "    <g id=\"text_17\">\n",
       "     <!-- Validation accuracy with noise channels -->\n",
       "     <g transform=\"translate(176.121875 214.000312)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 1831 0 \n",
       "L 50 4666 \n",
       "L 709 4666 \n",
       "L 2188 738 \n",
       "L 3669 4666 \n",
       "L 4325 4666 \n",
       "L 2547 0 \n",
       "L 1831 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-56\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 269 3500 \n",
       "L 844 3500 \n",
       "L 1563 769 \n",
       "L 2278 3500 \n",
       "L 2956 3500 \n",
       "L 3675 769 \n",
       "L 4391 3500 \n",
       "L 4966 3500 \n",
       "L 4050 0 \n",
       "L 3372 0 \n",
       "L 2619 2869 \n",
       "L 1863 0 \n",
       "L 1184 0 \n",
       "L 269 3500 \n",
       "z\n",
       "\" id=\"DejaVuSans-77\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-56\"/>\n",
       "      <use x=\"60.658203\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"121.9375\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use x=\"149.720703\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"177.503906\" xlink:href=\"#DejaVuSans-64\"/>\n",
       "      <use x=\"240.980469\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"302.259766\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use x=\"341.46875\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"369.251953\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"430.433594\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use x=\"493.8125\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"525.599609\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"586.878906\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"641.859375\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"696.839844\" xlink:href=\"#DejaVuSans-75\"/>\n",
       "      <use x=\"760.21875\" xlink:href=\"#DejaVuSans-72\"/>\n",
       "      <use x=\"801.332031\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"862.611328\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"917.591797\" xlink:href=\"#DejaVuSans-79\"/>\n",
       "      <use x=\"976.771484\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"1008.558594\" xlink:href=\"#DejaVuSans-77\"/>\n",
       "      <use x=\"1090.345703\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"1118.128906\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use x=\"1157.337891\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"1220.716797\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"1252.503906\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use x=\"1315.882812\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"1377.064453\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"1404.847656\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "      <use x=\"1456.947266\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use x=\"1518.470703\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"1550.257812\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"1605.238281\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"1668.617188\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"1729.896484\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use x=\"1793.275391\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use x=\"1856.654297\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use x=\"1918.177734\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use x=\"1945.960938\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_18\">\n",
       "     <path d=\"M 148.121875 225.178437 \n",
       "L 168.121875 225.178437 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_19\"/>\n",
       "    <g id=\"text_18\">\n",
       "     <!-- Validation accuracy with zeros channels -->\n",
       "     <g transform=\"translate(176.121875 228.678437)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 353 3500 \n",
       "L 3084 3500 \n",
       "L 3084 2975 \n",
       "L 922 459 \n",
       "L 3084 459 \n",
       "L 3084 0 \n",
       "L 275 0 \n",
       "L 275 525 \n",
       "L 2438 3041 \n",
       "L 353 3041 \n",
       "L 353 3500 \n",
       "z\n",
       "\" id=\"DejaVuSans-7a\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-56\"/>\n",
       "      <use x=\"60.658203\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"121.9375\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use x=\"149.720703\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"177.503906\" xlink:href=\"#DejaVuSans-64\"/>\n",
       "      <use x=\"240.980469\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"302.259766\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use x=\"341.46875\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"369.251953\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"430.433594\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use x=\"493.8125\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"525.599609\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"586.878906\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"641.859375\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"696.839844\" xlink:href=\"#DejaVuSans-75\"/>\n",
       "      <use x=\"760.21875\" xlink:href=\"#DejaVuSans-72\"/>\n",
       "      <use x=\"801.332031\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"862.611328\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"917.591797\" xlink:href=\"#DejaVuSans-79\"/>\n",
       "      <use x=\"976.771484\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"1008.558594\" xlink:href=\"#DejaVuSans-77\"/>\n",
       "      <use x=\"1090.345703\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"1118.128906\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use x=\"1157.337891\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"1220.716797\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"1252.503906\" xlink:href=\"#DejaVuSans-7a\"/>\n",
       "      <use x=\"1304.994141\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use x=\"1366.517578\" xlink:href=\"#DejaVuSans-72\"/>\n",
       "      <use x=\"1405.380859\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"1466.5625\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "      <use x=\"1518.662109\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"1550.449219\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"1605.429688\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"1668.808594\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"1730.087891\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use x=\"1793.466797\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use x=\"1856.845703\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use x=\"1918.369141\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use x=\"1946.152344\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p33b56e0cfd\">\n",
       "   <rect height=\"217.44\" width=\"334.8\" x=\"50.14375\" y=\"22.318125\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 710 ms (started: 2021-07-25 18:34:34 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "val_acc_noise = history_noise.history[\"val_accuracy\"]\n",
    "val_acc_zeros = history_zeros.history[\"val_accuracy\"]\n",
    "\n",
    "epochs = range(1, 11)\n",
    "\n",
    "plt.plot(epochs, val_acc_noise, \"b-\",\n",
    "label=\"Validation accuracy with noise channels\")\n",
    "plt.plot(epochs, val_acc_zeros, \"b--\",\n",
    "label=\"Validation accuracy with zeros channels\")\n",
    "\n",
    "plt.title(\"Effect of noise channels on validation accuracy\")\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Accuracy\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "entire-testament",
   "metadata": {},
   "source": [
    "尽管在两种情况下数据都包含相同的信息，但使用噪声通道训练的模型的验证准确度最终会降低约 1 个百分点——纯粹是由于虚假相关的影响。 您添加的噪声通道越多，精度将进一步降低。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "deluxe-casting",
   "metadata": {},
   "source": [
    "嘈杂的特征不可避免地会导致过度拟合。 因此，如果您不确定您拥有的特征是提供信息还是分散注意力，通常在训练前进行特征选择。 例如，将 IMDB 数据限制在前 10,000 个最常见的词是一种粗略的特征选择形式。 进行特征选择的典型方法是为每个可用特征计算一些有用性分数——衡量特征对任务的信息量，例如特征和标签之间的互信息——并且只保留那些可用的特征 高于某个阈值。 这样做会过滤掉上面例子中的白噪声通道。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "greater-idaho",
   "metadata": {},
   "source": [
    "### 深度学习中泛化的本质"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aging-airfare",
   "metadata": {},
   "source": [
    "关于深度学习模型的一个显着事实是，只要它们具有足够的表示能力，就可以训练它们以适应任何事物。\n",
    "\n",
    "不相信我？ 尝试改组 MNIST 标签并在其上训练模型。 即使输入和混洗后的标签之间没有任何关系，即使模型相对较小，训练损失也会下降得很好。 自然地，验证损失不会随着时间的推移而改善，因为在这种情况下没有泛化的可能性。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "distinguished-genius",
   "metadata": {},
   "source": [
    "> 清单 5.4 使用随机打乱的标签拟合 MNIST 模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "expected-viewer",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "375/375 [==============================] - 4s 8ms/step - loss: 2.3162 - accuracy: 0.1040 - val_loss: 2.3076 - val_accuracy: 0.1077\n",
      "Epoch 2/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 2.3005 - accuracy: 0.1187 - val_loss: 2.3097 - val_accuracy: 0.1119\n",
      "Epoch 3/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 2.2913 - accuracy: 0.1290 - val_loss: 2.3176 - val_accuracy: 0.1098\n",
      "Epoch 4/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 2.2778 - accuracy: 0.1415 - val_loss: 2.3211 - val_accuracy: 0.1014\n",
      "Epoch 5/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 2.2596 - accuracy: 0.1546 - val_loss: 2.3370 - val_accuracy: 0.1028\n",
      "Epoch 6/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 2.2389 - accuracy: 0.1682 - val_loss: 2.3556 - val_accuracy: 0.0985\n",
      "Epoch 7/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 2.2132 - accuracy: 0.1837 - val_loss: 2.3667 - val_accuracy: 0.1024\n",
      "Epoch 8/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 2.1862 - accuracy: 0.2006 - val_loss: 2.3785 - val_accuracy: 0.1020\n",
      "Epoch 9/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 2.1560 - accuracy: 0.2132 - val_loss: 2.4005 - val_accuracy: 0.0993\n",
      "Epoch 10/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 2.1236 - accuracy: 0.2333 - val_loss: 2.4172 - val_accuracy: 0.1018\n",
      "Epoch 11/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 2.0900 - accuracy: 0.2512 - val_loss: 2.4456 - val_accuracy: 0.1013\n",
      "Epoch 12/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 2.0546 - accuracy: 0.2646 - val_loss: 2.4901 - val_accuracy: 0.0987\n",
      "Epoch 13/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 2.0201 - accuracy: 0.2812 - val_loss: 2.5118 - val_accuracy: 0.0990\n",
      "Epoch 14/100\n",
      "375/375 [==============================] - 4s 9ms/step - loss: 1.9849 - accuracy: 0.2964 - val_loss: 2.5315 - val_accuracy: 0.0995\n",
      "Epoch 15/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 1.9496 - accuracy: 0.3100 - val_loss: 2.5727 - val_accuracy: 0.1047\n",
      "Epoch 16/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.9149 - accuracy: 0.3264 - val_loss: 2.5843 - val_accuracy: 0.1015\n",
      "Epoch 17/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.8788 - accuracy: 0.3411 - val_loss: 2.6309 - val_accuracy: 0.0960\n",
      "Epoch 18/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.8444 - accuracy: 0.3541 - val_loss: 2.6689 - val_accuracy: 0.0973\n",
      "Epoch 19/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.8098 - accuracy: 0.3649 - val_loss: 2.7240 - val_accuracy: 0.0997\n",
      "Epoch 20/100\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 1.7770 - accuracy: 0.3797 - val_loss: 2.7381 - val_accuracy: 0.0962\n",
      "Epoch 21/100\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 1.7453 - accuracy: 0.3942 - val_loss: 2.7705 - val_accuracy: 0.0992\n",
      "Epoch 22/100\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 1.7137 - accuracy: 0.4057 - val_loss: 2.8252 - val_accuracy: 0.0966\n",
      "Epoch 23/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.6820 - accuracy: 0.4178 - val_loss: 2.8518 - val_accuracy: 0.0951\n",
      "Epoch 24/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 1.6510 - accuracy: 0.4288 - val_loss: 2.9114 - val_accuracy: 0.0964\n",
      "Epoch 25/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 1.6219 - accuracy: 0.4419 - val_loss: 2.9385 - val_accuracy: 0.1039\n",
      "Epoch 26/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 1.5919 - accuracy: 0.4519 - val_loss: 2.9875 - val_accuracy: 0.0995\n",
      "Epoch 27/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 1.5650 - accuracy: 0.4589 - val_loss: 3.0283 - val_accuracy: 0.0968\n",
      "Epoch 28/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.5360 - accuracy: 0.4727 - val_loss: 3.0503 - val_accuracy: 0.0986\n",
      "Epoch 29/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.5094 - accuracy: 0.4815 - val_loss: 3.0967 - val_accuracy: 0.0990\n",
      "Epoch 30/100\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 1.4856 - accuracy: 0.4911 - val_loss: 3.1399 - val_accuracy: 0.0998\n",
      "Epoch 31/100\n",
      "375/375 [==============================] - 4s 10ms/step - loss: 1.4575 - accuracy: 0.5014 - val_loss: 3.2066 - val_accuracy: 0.1034\n",
      "Epoch 32/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 1.4335 - accuracy: 0.5105 - val_loss: 3.2479 - val_accuracy: 0.1017\n",
      "Epoch 33/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.4083 - accuracy: 0.5202 - val_loss: 3.3152 - val_accuracy: 0.0999\n",
      "Epoch 34/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 1.3850 - accuracy: 0.5276 - val_loss: 3.3304 - val_accuracy: 0.0975\n",
      "Epoch 35/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.3610 - accuracy: 0.5351 - val_loss: 3.3893 - val_accuracy: 0.0975\n",
      "Epoch 36/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.3368 - accuracy: 0.5461 - val_loss: 3.4373 - val_accuracy: 0.0965\n",
      "Epoch 37/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.3158 - accuracy: 0.5523 - val_loss: 3.4918 - val_accuracy: 0.1002\n",
      "Epoch 38/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 1.2925 - accuracy: 0.5613 - val_loss: 3.5276 - val_accuracy: 0.0996\n",
      "Epoch 39/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 1.2736 - accuracy: 0.5687 - val_loss: 3.5901 - val_accuracy: 0.1013\n",
      "Epoch 40/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 1.2512 - accuracy: 0.5767 - val_loss: 3.6409 - val_accuracy: 0.0978\n",
      "Epoch 41/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 1.2328 - accuracy: 0.5832 - val_loss: 3.7022 - val_accuracy: 0.0976\n",
      "Epoch 42/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.2086 - accuracy: 0.5905 - val_loss: 3.7671 - val_accuracy: 0.0962\n",
      "Epoch 43/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.1900 - accuracy: 0.5978 - val_loss: 3.7981 - val_accuracy: 0.0987\n",
      "Epoch 44/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.1674 - accuracy: 0.6043 - val_loss: 3.8791 - val_accuracy: 0.0971\n",
      "Epoch 45/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 1.1506 - accuracy: 0.6145 - val_loss: 3.8802 - val_accuracy: 0.1000\n",
      "Epoch 46/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.1320 - accuracy: 0.6202 - val_loss: 3.9443 - val_accuracy: 0.0990\n",
      "Epoch 47/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.1128 - accuracy: 0.6249 - val_loss: 4.0107 - val_accuracy: 0.0987\n",
      "Epoch 48/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.0947 - accuracy: 0.6315 - val_loss: 4.0480 - val_accuracy: 0.0975\n",
      "Epoch 49/100\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 1.0781 - accuracy: 0.6389 - val_loss: 4.1215 - val_accuracy: 0.0995\n",
      "Epoch 50/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.0599 - accuracy: 0.6461 - val_loss: 4.1831 - val_accuracy: 0.1004\n",
      "Epoch 51/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.0469 - accuracy: 0.6489 - val_loss: 4.2270 - val_accuracy: 0.1017\n",
      "Epoch 52/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 1.0284 - accuracy: 0.6565 - val_loss: 4.3085 - val_accuracy: 0.1012\n",
      "Epoch 53/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.0125 - accuracy: 0.6619 - val_loss: 4.3652 - val_accuracy: 0.0967\n",
      "Epoch 54/100\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 0.9961 - accuracy: 0.6689 - val_loss: 4.4272 - val_accuracy: 0.1023\n",
      "Epoch 55/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.9811 - accuracy: 0.6737 - val_loss: 4.4548 - val_accuracy: 0.1002\n",
      "Epoch 56/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.9649 - accuracy: 0.6793 - val_loss: 4.5626 - val_accuracy: 0.0996\n",
      "Epoch 57/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.9522 - accuracy: 0.6820 - val_loss: 4.5705 - val_accuracy: 0.1003\n",
      "Epoch 58/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.9359 - accuracy: 0.6894 - val_loss: 4.6338 - val_accuracy: 0.0975\n",
      "Epoch 59/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.9218 - accuracy: 0.6924 - val_loss: 4.7335 - val_accuracy: 0.0974\n",
      "Epoch 60/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.9077 - accuracy: 0.7002 - val_loss: 4.7490 - val_accuracy: 0.1002\n",
      "Epoch 61/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.8919 - accuracy: 0.7061 - val_loss: 4.8261 - val_accuracy: 0.0968\n",
      "Epoch 62/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.8792 - accuracy: 0.7084 - val_loss: 4.8828 - val_accuracy: 0.0992\n",
      "Epoch 63/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.8646 - accuracy: 0.7143 - val_loss: 4.9364 - val_accuracy: 0.1003\n",
      "Epoch 64/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.8548 - accuracy: 0.7187 - val_loss: 4.9928 - val_accuracy: 0.0989\n",
      "Epoch 65/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.8393 - accuracy: 0.7230 - val_loss: 5.0592 - val_accuracy: 0.0997\n",
      "Epoch 66/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.8278 - accuracy: 0.7281 - val_loss: 5.1535 - val_accuracy: 0.1018\n",
      "Epoch 67/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.8175 - accuracy: 0.7283 - val_loss: 5.2259 - val_accuracy: 0.0936\n",
      "Epoch 68/100\n",
      "375/375 [==============================] - 4s 11ms/step - loss: 0.8013 - accuracy: 0.7348 - val_loss: 5.2622 - val_accuracy: 0.0968\n",
      "Epoch 69/100\n",
      "375/375 [==============================] - 4s 10ms/step - loss: 0.7916 - accuracy: 0.7404 - val_loss: 5.3013 - val_accuracy: 0.0974\n",
      "Epoch 70/100\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 0.7774 - accuracy: 0.7445 - val_loss: 5.3868 - val_accuracy: 0.0991\n",
      "Epoch 71/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.7670 - accuracy: 0.7461 - val_loss: 5.4737 - val_accuracy: 0.0968\n",
      "Epoch 72/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.7561 - accuracy: 0.7510 - val_loss: 5.4647 - val_accuracy: 0.0984\n",
      "Epoch 73/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.7435 - accuracy: 0.7561 - val_loss: 5.5492 - val_accuracy: 0.0988\n",
      "Epoch 74/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.7336 - accuracy: 0.7595 - val_loss: 5.6337 - val_accuracy: 0.0962\n",
      "Epoch 75/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.7230 - accuracy: 0.7633 - val_loss: 5.7366 - val_accuracy: 0.0961\n",
      "Epoch 76/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.7125 - accuracy: 0.7673 - val_loss: 5.7615 - val_accuracy: 0.1008\n",
      "Epoch 77/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.7019 - accuracy: 0.7697 - val_loss: 5.8639 - val_accuracy: 0.0983\n",
      "Epoch 78/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.6927 - accuracy: 0.7704 - val_loss: 5.8943 - val_accuracy: 0.0986\n",
      "Epoch 79/100\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 0.6808 - accuracy: 0.7774 - val_loss: 5.9849 - val_accuracy: 0.0979\n",
      "Epoch 80/100\n",
      "375/375 [==============================] - 4s 10ms/step - loss: 0.6715 - accuracy: 0.7817 - val_loss: 5.9962 - val_accuracy: 0.0970\n",
      "Epoch 81/100\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 0.6613 - accuracy: 0.7846 - val_loss: 6.1039 - val_accuracy: 0.0963\n",
      "Epoch 82/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.6550 - accuracy: 0.7862 - val_loss: 6.1493 - val_accuracy: 0.0978\n",
      "Epoch 83/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.6437 - accuracy: 0.7910 - val_loss: 6.2056 - val_accuracy: 0.0983\n",
      "Epoch 84/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.6340 - accuracy: 0.7924 - val_loss: 6.2864 - val_accuracy: 0.0968\n",
      "Epoch 85/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.6233 - accuracy: 0.7988 - val_loss: 6.3221 - val_accuracy: 0.0978\n",
      "Epoch 86/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.6182 - accuracy: 0.7986 - val_loss: 6.4431 - val_accuracy: 0.0947\n",
      "Epoch 87/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.6072 - accuracy: 0.8034 - val_loss: 6.4944 - val_accuracy: 0.0983\n",
      "Epoch 88/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.6002 - accuracy: 0.8042 - val_loss: 6.5432 - val_accuracy: 0.0971\n",
      "Epoch 89/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.5901 - accuracy: 0.8094 - val_loss: 6.5760 - val_accuracy: 0.0950\n",
      "Epoch 90/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.5833 - accuracy: 0.8110 - val_loss: 6.7064 - val_accuracy: 0.0975\n",
      "Epoch 91/100\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 0.5739 - accuracy: 0.8148 - val_loss: 6.6991 - val_accuracy: 0.1012\n",
      "Epoch 92/100\n",
      "375/375 [==============================] - 4s 10ms/step - loss: 0.5644 - accuracy: 0.8166 - val_loss: 6.8536 - val_accuracy: 0.0986\n",
      "Epoch 93/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.5564 - accuracy: 0.8196 - val_loss: 6.8880 - val_accuracy: 0.0972\n",
      "Epoch 94/100\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.5521 - accuracy: 0.8218 - val_loss: 6.9333 - val_accuracy: 0.0965\n",
      "Epoch 95/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.5426 - accuracy: 0.8245 - val_loss: 7.0386 - val_accuracy: 0.0978\n",
      "Epoch 96/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.5337 - accuracy: 0.8285 - val_loss: 7.1125 - val_accuracy: 0.0961\n",
      "Epoch 97/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.5274 - accuracy: 0.8294 - val_loss: 7.1403 - val_accuracy: 0.0993\n",
      "Epoch 98/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.5194 - accuracy: 0.8325 - val_loss: 7.1998 - val_accuracy: 0.0978\n",
      "Epoch 99/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.5149 - accuracy: 0.8342 - val_loss: 7.2759 - val_accuracy: 0.0938\n",
      "Epoch 100/100\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.5053 - accuracy: 0.8378 - val_loss: 7.3778 - val_accuracy: 0.0969\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f85a80bce50>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 4min 57s (started: 2021-07-25 18:40:26 +08:00)\n"
     ]
    }
   ],
   "source": [
    "(train_images, train_labels), _ = mnist.load_data()\n",
    "\n",
    "train_images = train_images.reshape((60000, 28 * 28))\n",
    "train_images = train_images.astype(\"float32\") / 255\n",
    "\n",
    "random_train_labels = train_labels[:]\n",
    "\n",
    "np.random.shuffle(random_train_labels)\n",
    "\n",
    "model = keras.Sequential([\n",
    " layers.Dense(512, activation=\"relu\"),\n",
    " layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"sparse_categorical_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "model.fit(train_images, random_train_labels,\n",
    "          epochs=100,\n",
    "          batch_size=128,\n",
    "          validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "italic-forestry",
   "metadata": {},
   "source": [
    "事实上，你甚至不需要用 MNIST 数据来做这件事——你可以只生成白噪声输入和随机标签。 您也可以在其上拟合模型，只要它有足够的参数。 它最终只会记住特定的输入，就像 Python 字典一样。\n",
    "\n",
    "如果是这种情况，那么深度学习模型如何泛化呢？ 他们不应该只是学习训练输入和目标之间的临时映射，就像一个幻想吗？ 我们可以期望这个映射适用于新输入吗？\n",
    "\n",
    "事实证明，深度学习中泛化的本质与深度学习模型本身关系不大，而与现实世界中的信息结构关系很大。 让我们来看看这里到底发生了什么。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "destroyed-february",
   "metadata": {},
   "source": [
    "**流形假说**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "downtown-racing",
   "metadata": {},
   "source": [
    "MNIST 分类器的输入（在预处理之前）是一个 28x28 的 0 到 255 之间的整数数组。因此可能输入值的总数是 256 的 784 次方——远大于宇宙中原子的数量。 然而，这些输入中很少有看起来像有效的 MNIST 样本：实际的手写数字仅占用所有可能的 28x28 uint8 数组的父子空间空间的一小部分。 更重要的是，这个子空间不仅仅是在父空间中随机散布的一组点：它是高度结构化的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "quality-study",
   "metadata": {},
   "source": [
    "首先，有效手写数字的子空间是：如果你取一个样本并对其连续修改一点，它仍然可以识别为同一个手写数字。 此外，有效子空间中的所有样本都是通过穿过子空间的平滑路径。 这意味着如果你连接了两个随机的 MNIST 数字 A 和 B，就会存在一系列“中间”图像，将 A 变形为 B，这样两个连续的数字彼此非常接近（见图 5.7）。 也许在两个类之间的边界附近会有一些模糊的形状，但即使是这些形状看起来仍然很像数字。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "joined-lesbian",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gstdzkq2yzj31bk0j6win.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "established-attraction",
   "metadata": {},
   "source": [
    "在技术术语中，您会说手写数字在多种可能的 28x28 uint8 数组的空间内形成 a。 这是一个很大的词，但这个概念非常直观。 “流形”是某个父空间的低维子空间，它在局部类似于线性（欧几里得）空间。 例如，平面中的平滑曲线是二维空间中的一维流形，因为对于曲线的每个点，您都可以绘制一条切线（曲线可以在每个点上由一条线近似）。 3D 空间内的光滑表面是 2D 流形。 等等。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "intermediate-miller",
   "metadata": {},
   "source": [
    "更一般地说，流形假设假定所有自然数据都位于编码它的高维空间内的低维流形上。 这是关于宇宙信息结构的一个非常有力的陈述。 据我们所知，它是准确的，这也是深度学习起作用的原因。 对于 MNIST 数字如此，对于人脸、树木形态、人声甚至自然语言也是如此。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "executed-titanium",
   "metadata": {},
   "source": [
    "流形假设意味着：\n",
    "* 机器学习模型只需要在其潜在的输入空间（潜在流形）中拟合相对简单、低维、高度结构化的子空间。\n",
    "* 在这些流形之一中，始终可以在两个输入之间进行插值，也就是说，通过所有点都落在流形上的连续路径将一个输入变形为另一个。\n",
    "\n",
    "在样本之间进行插值的能力是理解深度学习泛化的关键。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "municipal-retailer",
   "metadata": {},
   "source": [
    "**插值作为泛化源**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bizarre-mailman",
   "metadata": {},
   "source": [
    "如果您使用可以插值的数据点，您可以开始理解以前从未见过的点，通过将它们与位于流形上的其他点相关联。 换句话说，您可以仅使用空间样本来理解空间的意义。 您的总体样本可以使用插值来填补空白。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "painted-senior",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gste6o6usij31cu0o0gob.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hydraulic-rocket",
   "metadata": {},
   "source": [
    "请注意，潜在流形上的插值与父空间中的线性插值不同，如图 5.8 所示。 例如，两个 MNIST 数字之间的像素平均值通常不是有效数字。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fifth-commitment",
   "metadata": {},
   "source": [
    "至关重要的是，虽然深度学习通过对数据流形的学习近似值进行插值来实现泛化，但假设插值就是泛化的全部内容是错误的。 这是冰山一角。 插值只能帮助您理解与您以前见过的非常接近的事物：它支持局部泛化。 但值得注意的是，人类一直在处理极端新奇事物，而且他们做得很好。 您无需事先接受有关您将不得不遇到的每种情况的无数示例的培训。 你的每一天都不同于你以前经历过的任何一天，也不同于自人类诞生以来任何人经历过的任何一天。 您可以在纽约一周、上海一周和班加罗尔一周之间切换，而无需为每个城市进行数千次的学习和排练。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "solar-faculty",
   "metadata": {},
   "source": [
    "人类能够进行极端概括，这是通过插值以外的认知机制实现的——抽象、世界的符号模型、推理、逻辑、常识、关于世界的先天先验。 我们通常所说的，与直觉和模式原因识别相反。 后者本质上在很大程度上是内插的，但前者不是。 两者对于智能都是必不可少的。 我们将在第 14 章详细讨论这一点。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "excited-boston",
   "metadata": {},
   "source": [
    "**为什么深度学习有效**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "experienced-authentication",
   "metadata": {},
   "source": [
    "还记得第 2 章中皱巴巴的纸球比喻吗？ 一张纸代表 3D 空间中的 2D 流形。 深度学习模型是一种用于解开纸团的工具，即用于解开潜在流形的工具。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "clear-revolution",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gsteaodbtsj31dk0ikjt7.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "certified-pressing",
   "metadata": {},
   "source": [
    "深度学习模型基本上是一条非常高维的曲线。 一条平滑且连续的曲线（对其结构有额外的约束，源自模型架构先验），因为它需要可微。 并且该曲线通过梯度下降（平滑且渐进）拟合到数据点。 通过构建，深度学习是采用一条大而复杂的曲线——一个流形——并逐步调整其参数，直到它适合一些训练数据点。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "common-regard",
   "metadata": {},
   "source": [
    "曲线包含足够多的参数，可以拟合任何东西——事实上，如果你让模型训练足够长的时间，它最终会有效地完全记住它的训练数据，根本不会泛化。 但是，您拟合的数据并不是由在底层空间中稀疏分布的孤立点组成的。 您的数据在输入空间内形成了一个高度结构化、低维的流形——这就是流形假设。 并且因为模型曲线与这些数据的拟合会随着时间的推移逐渐平滑地发生，随着梯度下降的进行，在训练过程中会有一个中间点，在该点模型大致接近数据的自然流形，如图 5.10 所示。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "systematic-station",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gsted0w8lpj31es0octd2.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "framed-microphone",
   "metadata": {},
   "source": [
    "沿着模型在该点学习的曲线移动将接近沿着数据的实际潜在流形移动——因此，模型将能够通过训练输入之间的插值来理解从未见过的输入。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "brazilian-bones",
   "metadata": {},
   "source": [
    "除了它们具有足够的表示能力这一微不足道的事实之外，深度学习模型还有一些特性使它们特别适合学习潜在流形：\n",
    "1. 深度学习模型实现了从输入到输出的平滑、连续映射。 它必须是平滑和连续的，因为它必须是可微的（否则你不能做梯度下降）。 这种平滑性有助于逼近具有相同属性的潜在流形。\n",
    "2. 深度学习模型的结构往往反映了训练数据中信息的“形状”（通过架构先验）。 对于图像处理模型（参见第 8 章和第 9 章）和序列处理模型（参见第 10 章）尤其如此。 更一般地说，深度神经网络以分层和模块化的方式构建他们学习的表示，这与自然数据的组织方式相呼应。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "invalid-alias",
   "metadata": {},
   "source": [
    "**训练数据至关重要**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "coordinated-ridge",
   "metadata": {},
   "source": [
    "虽然深度学习确实非常适合流形学习，但泛化能力更多是数据自然结构的结果，而不是模型任何属性的结果。 只有当您的数据形成可以插入点的流形时，您才能进行概括。 你的特征信息量越多，噪音越小，你的概括能力就越好，因为你的输入空间会更简单，结构也更好。 数据管理和特征工程对于泛化至关重要。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "departmental-louisville",
   "metadata": {},
   "source": [
    "此外，由于深度学习是曲线拟合，为了使模型表现良好，需要对其输入空间的密集采样进行训练。 在这种情况下，“密集采样”意味着训练数据应该密集地覆盖整个输入数据流形（见图 5.11）。 在决策边界附近尤其如此。 通过足够密集的采样，可以通过在过去的训练输入之间进行插值来理解新输入，而无需使用常识、抽象推理或关于世界的外部知识——机器学习模型无法访问的所有事物。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "demanding-casino",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gstejo1o81j31ag0r641p.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "deluxe-belize",
   "metadata": {},
   "source": [
    "因此，您应该始终牢记，改进深度学习模型的最佳方法是在更多数据或更好的数据上对其进行训练（当然，添加过于嘈杂或不准确的数据会损害泛化）。 输入数据流形的更密集覆盖将产生一个更好泛化的模型。 您永远不应该期望深度学习模型在其训练样本之间执行任何粗略的插值，因此，您应该尽一切可能使插值尽可能简单。 你会在深度学习模型中找到的唯一东西就是你放入其中的内容：在其架构中编码的先验，以及用于训练的数据。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "economic-congo",
   "metadata": {},
   "source": [
    "当无法获取更多数据时，次优解决方案是调整模型允许存储的信息量，或者对模型曲线的平滑度添加约束。 如果网络只能记住少量模式或非常规则的模式，优化过程将迫使它专注于最突出的模式，这些模式有更好的泛化机会。 以这种方式对抗过拟合的处理称为正则化。 我们将在 5.4.4 节深入回顾正则化技术。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adapted-modeling",
   "metadata": {},
   "source": [
    "在开始调整模型以帮助它更好地泛化之前，您需要一种方法来评估模型当前的表现。 在下一节中，您将了解如何在模型开发过程中监控泛化：模型评估。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "female-multimedia",
   "metadata": {},
   "source": [
    "## 评估机器学习模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "initial-demand",
   "metadata": {},
   "source": [
    "你只能控制你能观察到的东西。 由于您的目标是开发可以成功泛化到新数据的模型，因此能够可靠地衡量模型的泛化能力至关重要。 在本节中，我们将正式介绍评估机器学习模型的不同方法。 您已经在上一章中看到了其中的大部分。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "impaired-circumstances",
   "metadata": {},
   "source": [
    "### 训练、验证和测试集"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "signed-message",
   "metadata": {},
   "source": [
    "评估模型总是归结为将可用数据分成三组：训练、验证和测试。 您在训练数据上进行训练并在验证数据上评估您的模型。 一旦您的模型准备好迎接黄金时段，您就可以在测试数据上对其进行最后一次测试，这意味着尽可能与生产数据相似。 然后您可以在生产中部署模型。\n",
    "\n",
    "你可能会问，为什么不用两个集合：一个训练集和一个测试集？ 您将在训练数据上进行训练并在测试数据上进行评估。 简单多了！"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "western-dublin",
   "metadata": {},
   "source": [
    "原因是开发模型总是涉及调整其配置：例如，选择层数或层的大小（称为模型的超参数，以区别于 ，后者是网络的权重）。 您可以通过使用模型在验证数据上的性能作为反馈信号来调整参数。 本质上，这种调整是一种形式：在某些参数空间中搜索良好的配置。 作为学习结果，根据模型在验证集上的性能调整模型的配置可能会很快导致对验证集的过度拟合，即使您的模型从未直接对其进行过训练。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "subject-cooking",
   "metadata": {},
   "source": [
    "这种现象的核心是信息泄漏的概念。 每次根据模型在验证集上的性能调整模型的超参数时，有关验证数据的一些信息就会泄漏到模型中。 如果您只对一个参数执行一次此操作，那么将泄漏很少的信息位，并且您的验证集将保持可靠以评估模型。 但是如果你重复这个过程很多次——运行一个实验，评估验证集，然后修改你的模型——那么你就会将越来越多的关于验证集的信息泄漏到模型中。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "normal-superintendent",
   "metadata": {},
   "source": [
    "在一天结束时，您最终会得到一个在验证数据上人为执行良好的模型，因为这就是您对其进行优化的目的。 您关心全新的性能\n",
    "数据，而不是验证数据，因此您需要使用完全不同的、前所未见的数据集来评估模型：测试数据集。 您的模型不应该访问任何信息\n",
    "关于测试集，甚至是间接的。 如果根据测试集性能调整了有关模型的任何内容，那么您的泛化度量将有缺陷。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "worldwide-retail",
   "metadata": {},
   "source": [
    "将数据拆分为训练、验证和测试集似乎很简单，但有一些高级方法可以在数据很少时派上用场。 让我们回顾一下三个经典的评估方法：简单的保持验证、K 折验证和带混洗的迭代 K 折验证。 我们还将讨论使用常识基线来检查您的训练是否在某个地方进行。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "built-lunch",
   "metadata": {},
   "source": [
    "**简单的保持验证**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "official-netherlands",
   "metadata": {},
   "source": [
    "将数据的一部分作为测试集。 对剩余数据进行训练，并在测试集上进行评估。 正如您在前几节中看到的，为了防止信息泄漏，您不应该根据测试集调整模型，因此您还应该保留一个验证集。\n",
    "\n",
    "示意性地，保持验证如图 5.12 所示。 下面的清单显示了一个简单的实现。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "durable-ocean",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gsteu6kivvj31g00sc40k.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exotic-methodology",
   "metadata": {},
   "source": [
    "> 清单 5.5 Hold-out 验证（注意为了简单起见省略了标签）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "honest-republican",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_validation_samples = 10000\n",
    "np.random.shuffle(data)\n",
    "\n",
    "validation_data = data[:num_validation_samples]\n",
    "training_data = data[num_validation_samples:]\n",
    "\n",
    "model = get_model()\n",
    "\n",
    "model.fit(training_data, ...)\n",
    "\n",
    "validation_score = model.evaluate(validation_data, ...) \n",
    "\n",
    "...\n",
    "model = get_model()\n",
    "model.fit(np.concatenate([training_data,\n",
    "                          validation_data]), \n",
    "          ...) \n",
    "test_score = model.evaluate(test_data, ...)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wooden-baghdad",
   "metadata": {},
   "source": [
    "这是最简单的评估协议，它有一个缺陷：如果可用数据很少，那么您的验证和测试集可能包含的样本太少，无法在统计上代表手头的数据。 这很容易识别：如果在拆分之前对数据进行不同的随机混洗轮最终会产生非常不同的模型性能度量，那么您就会遇到这个问题。 K 折验证和迭代 K 折验证是解决此问题的两种方法，如下所述。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "incredible-zealand",
   "metadata": {},
   "source": [
    "**K 折验证**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dominican-modem",
   "metadata": {},
   "source": [
    "使用这种方法，您可以将数据拆分为大小相等的分区。 对于每个分区 i，在剩余分区上训练一个 K-1 模型，并在分区 上对其进行评估。 您的最终分数是所获得的 K 分数的平均值。 当您的模型性能根据您的训练测试拆分显示出显着差异时，此方法很有用。 与保留验证一样，此方法并不能免除您使用不同的验证集进行模型校准。 \n",
    "\n",
    "K 折交叉验证的示意图如图 5.13 所示。 清单 5.6 展示了一个简单的实现。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "healthy-filling",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gsteyc3stgj31cc0lmgnq.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "breeding-damages",
   "metadata": {},
   "source": [
    "> 清单 5.6 K 折交叉验证（注意为简单起见省略了标签）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "particular-shannon",
   "metadata": {},
   "outputs": [],
   "source": [
    "k = 3\n",
    "num_validation_samples = len(data) // k\n",
    "np.random.shuffle(data)\n",
    "validation_scores = []\n",
    "\n",
    "for fold in range(k):\n",
    "    validation_data = data[num_validation_samples * fold:num_validation_samples * (fold + 1)] \n",
    "    training_data = np.concatenate(data[:num_validation_samples * fold],\n",
    "                                   data[num_validation_samples * (fold + 1):])\n",
    "    model = get_model()\n",
    "    model.fit(training_data, ...)\n",
    "    validation_score = model.evaluate(validation_data, ...)\n",
    "    validation_scores.append(validation_score)\n",
    "        \n",
    "validation_score = np.average(validation_scores)\n",
    "model = get_model()\n",
    "model.fit(data, ...)\n",
    "test_score = model.evaluate(test_data, ...)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "important-sweet",
   "metadata": {},
   "source": [
    "**带洗牌的迭代 K 折验证**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "imported-retention",
   "metadata": {},
   "source": [
    "这适用于可用数据相对较少且需要尽可能精确地评估模型的情况。 我发现它在 Kaggle 比赛中非常有帮助。 它包括多次应用 K 折验证，每次在将数据拆分 K 种方式之前对数据进行混洗。 最终分数是每次运行 K 折验证时获得的分数的平均值。 请注意，您最终会训练和评估 P * K 模型（其中 P 是您使用的迭代次数），这可能非常昂贵。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "traditional-programmer",
   "metadata": {},
   "source": [
    "### 击败常识基线"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aerial-portal",
   "metadata": {},
   "source": [
    "除了您可用的不同评估协议之外，您还应该了解的最后一件事是使用常识基线。\n",
    "\n",
    "训练深度学习模型有点像在平行世界按下按钮发射火箭。 你听不到也看不到。 你无法观察流形学习过程——它发生在一个有数千个维度的空间中，即使你将它投影到 3D，你也无法解释它。 您获得的唯一反馈是您的验证指标——例如隐形火箭上的高度计。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "plain-tobago",
   "metadata": {},
   "source": [
    "特别重要的一点是能够判断您是否完全脱离了地面。 你开始的高度是多少？ 你的模型似乎有 15% 的准确率，这有什么好处吗？ 在开始使用数据集之前，您应该始终选择一个您将尝试击败的微不足道的基线。 如果你超过了这个门槛，你就会知道你做对了：你的模型实际上是在使用输入数据中的信息来进行泛化预测——你可以继续前进。 该基线可以是随机分类器的性能，也可以是您可以想象的最简单的非机器学习技术的性能。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "local-melissa",
   "metadata": {},
   "source": [
    "例如，在 MNIST 数字分类示例中，一个简单的基线是验证准确度大于 0.1（随机分类器）； 在 IMDB 示例中，验证准确度将大于 0.5。 在路透社的例子中，由于类别不平衡，它将在 0.18-0.19 左右。 如果您有一个二元分类问题，其中 90% 的样本属于 A 类，10% 属于 B 类，那么始终预测 A 的分类器的验证准确率已经达到 0.9，您需要做得比这更好。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beautiful-fifteen",
   "metadata": {},
   "source": [
    "当你开始解决一个以前没有人解决过的问题时，拥有一个可以参考的常识基线是必不可少的。 如果您无法击败一个微不足道的解决方案，那么您的模型就毫无价值——也许您使用了错误的模型，或者您正在解决的问题甚至无法首先通过机器学习来解决。 是时候回到绘图板了。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "raised-counter",
   "metadata": {},
   "source": [
    "### 关于模型评估的注意事项"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lovely-scheme",
   "metadata": {},
   "source": [
    "在选择评估协议时，请注意以下几点：\n",
    "* **数据代表性**——你希望你的训练集和测试集都能代表手头的数据。 例如，如果您尝试对数字图像进行分类，并且您从一个样本数组开始，其中样本按类别排序，则将数组的前 80% 作为训练集，其余 20% 因为您的测试集将导致您的训练集仅包含 0-7 类，而您的测试集仅包含 8-9 类。 这似乎是一个荒谬的错误，但它却出奇地普遍。 出于这个原因，您通常应该在将数据拆分为训练集和测试集之前随机打乱数据。\n",
    "* **时间之箭**——如果你想根据过去预测未来（例如，明天的天气、股票走势等），你不应该在拆分数据之前随机打乱数据，因为这样做会创建一个时间序列 泄漏：您的模型将有效地接受未来数据的训练。 在这种情况下，您应该始终确保测试集中的所有数据都对应于训练集中的数据。\n",
    "* **数据中的冗余**——如果数据中的某些数据点出现两次（在现实世界中很常见），那么将数据打乱并将其拆分为训练集和验证集将导致训练集和验证集之间出现冗余。 实际上，您将测试部分训练数据，这是您能做的最糟糕的事情！ 确保您的训练集和验证集不相交。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "duplicate-estonia",
   "metadata": {},
   "source": [
    "有一种可靠的方法来评估模型的性能，是您如何能够监控机器学习核心的张力——优化和泛化、欠拟合和过拟合。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "compressed-bridge",
   "metadata": {},
   "source": [
    "## 改进模型拟合"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "typical-palmer",
   "metadata": {},
   "source": [
    "为了达到完美的拟合，你必须首先过拟合。 既然你事先不知道边界在哪里，你必须越过它才能找到它。 因此，当您开始解决问题时，您的初始目标是实现一个模型，该模型具有一定的泛化能力，并且能够过拟合。 一旦你有了这样的模型，你将专注于通过对抗过度拟合来完善泛化。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "foreign-spine",
   "metadata": {},
   "source": [
    "在这个阶段你会遇到三个常见的问题：\n",
    "* 训练没有开始：你的训练损失不会随着时间的推移而减少。\n",
    "* 训练开始得很好，但你的模型没有有意义的概括：你无法击败你设置的常识基线。\n",
    "* 训练和验证损失都随着时间的推移而下降，你可以超过基线，但你似乎无法过拟合——这表明你仍然欠拟合。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "injured-annual",
   "metadata": {},
   "source": [
    "让我们看看如何解决这些问题以实现机器学习项目的第一个重要里程碑：获得一个具有一定泛化能力（它可以击败微不足道的基线）并且能够过度拟合的模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "destroyed-aquatic",
   "metadata": {},
   "source": [
    "### 调整关键梯度下降参数"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "inside-royal",
   "metadata": {},
   "source": [
    "有时，培训没有开始，或者过早停止。 你的损失被卡住了。 这始终是您可以克服的：请记住，您可以将模型拟合到随机数据。 即使你的问题没有任何意义，你也应该能够训练一些东西——只要仍然记住训练数据。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "photographic-survey",
   "metadata": {},
   "source": [
    "发生这种情况时，梯度下降过程的配置始终存在问题：您选择的优化器、模型权重的初始值分布、学习率或批量大小。 所有这些参数都是相互依赖的，因此，在保持其余参数不变的同时调整学习率和批量大小通常就足够了。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "danish-spank",
   "metadata": {},
   "source": [
    "让我们看一个具体的例子：让我们训练第 2 章中的 MNIST 模型，使用不恰当的大学习率，值为 1。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "variable-cancellation",
   "metadata": {},
   "source": [
    "> 清单 5.7 以错误的高学习率训练 MNIST 模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "aggregate-developer",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 27.7234 - accuracy: 0.1900 - val_loss: 2.1154 - val_accuracy: 0.2252\n",
      "Epoch 2/10\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 2.3795 - accuracy: 0.2013 - val_loss: 2.0841 - val_accuracy: 0.2294\n",
      "Epoch 3/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 2.2546 - accuracy: 0.2023 - val_loss: 1.9762 - val_accuracy: 0.2538\n",
      "Epoch 4/10\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 2.1286 - accuracy: 0.2598 - val_loss: 1.7637 - val_accuracy: 0.3400\n",
      "Epoch 5/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.9951 - accuracy: 0.2859 - val_loss: 1.7126 - val_accuracy: 0.3251\n",
      "Epoch 6/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.9974 - accuracy: 0.2924 - val_loss: 2.6586 - val_accuracy: 0.2213\n",
      "Epoch 7/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.9916 - accuracy: 0.2870 - val_loss: 2.0091 - val_accuracy: 0.3162\n",
      "Epoch 8/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.9645 - accuracy: 0.2970 - val_loss: 2.5020 - val_accuracy: 0.2513\n",
      "Epoch 9/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 1.9957 - accuracy: 0.2968 - val_loss: 2.4142 - val_accuracy: 0.2488\n",
      "Epoch 10/10\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 1.9588 - accuracy: 0.3098 - val_loss: 2.2652 - val_accuracy: 0.2592\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f89705023d0>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 29.7 s (started: 2021-07-25 20:53:24 +08:00)\n"
     ]
    }
   ],
   "source": [
    "train_images = train_images.reshape((60000, 28 * 28))\n",
    "train_images = train_images.astype(\"float32\") / 255\n",
    "\n",
    "model = keras.Sequential([\n",
    "    layers.Dense(512, activation=\"relu\"),\n",
    "    layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "\n",
    "model.compile(optimizer=keras.optimizers.RMSprop(1.),\n",
    "              loss=\"sparse_categorical_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "model.fit(train_images, \n",
    "          train_labels,\n",
    "          epochs=10,\n",
    "          batch_size=128,\n",
    "          validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "iraqi-telescope",
   "metadata": {},
   "source": [
    "该模型很快达到了 30%-40% 范围内的训练和验证准确率，但无法超过这个范围。 让我们尝试将学习率降低到更合理的值 1e-2："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "handy-illness",
   "metadata": {},
   "source": [
    "> 清单 5.8 具有更合适学习率的相同模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "sublime-florida",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "375/375 [==============================] - 4s 9ms/step - loss: 0.3486 - accuracy: 0.9128 - val_loss: 0.1742 - val_accuracy: 0.9531\n",
      "Epoch 2/10\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.1443 - accuracy: 0.9637 - val_loss: 0.1606 - val_accuracy: 0.9638\n",
      "Epoch 3/10\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.1166 - accuracy: 0.9727 - val_loss: 0.2224 - val_accuracy: 0.9644\n",
      "Epoch 4/10\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.1056 - accuracy: 0.9783 - val_loss: 0.1895 - val_accuracy: 0.9709\n",
      "Epoch 5/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0870 - accuracy: 0.9811 - val_loss: 0.2133 - val_accuracy: 0.9695\n",
      "Epoch 6/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0790 - accuracy: 0.9847 - val_loss: 0.2298 - val_accuracy: 0.9722\n",
      "Epoch 7/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0690 - accuracy: 0.9870 - val_loss: 0.2510 - val_accuracy: 0.9722\n",
      "Epoch 8/10\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0691 - accuracy: 0.9877 - val_loss: 0.2702 - val_accuracy: 0.9732\n",
      "Epoch 9/10\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 0.0596 - accuracy: 0.9895 - val_loss: 0.3145 - val_accuracy: 0.9728\n",
      "Epoch 10/10\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0578 - accuracy: 0.9900 - val_loss: 0.3171 - val_accuracy: 0.9719\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f897063ca00>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 30 s (started: 2021-07-25 20:52:25 +08:00)\n"
     ]
    }
   ],
   "source": [
    "model = keras.Sequential([\n",
    "    layers.Dense(512, activation=\"relu\"),\n",
    "    layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "\n",
    "model.compile(optimizer=keras.optimizers.RMSprop(1e-2),\n",
    "              loss=\"sparse_categorical_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "model.fit(train_images, \n",
    "          train_labels,\n",
    "          epochs=10,\n",
    "          batch_size=128,\n",
    "          validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beginning-message",
   "metadata": {},
   "source": [
    "该模型现在可以训练了。\n",
    "\n",
    "如果您发现自己处于类似情况，请尝试：\n",
    "* 降低或提高学习率。 太高的学习率可能会导致更新大大超过适当的拟合，就像上面的例子一样，而太低的学习率可能会使训练变得如此缓慢，以至于它似乎停滞不前。\n",
    "* 增加批量大小。 具有更多样本的批次将导致梯度信息更多且噪声更少（方差更低）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ignored-stone",
   "metadata": {},
   "source": [
    "最终，您将找到一个可以开始训练的配置。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "controversial-stage",
   "metadata": {},
   "source": [
    "### 利用更好的先验架构"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vital-winner",
   "metadata": {},
   "source": [
    "您有一个适合的模型，但由于某种原因，您的验证指标根本没有改善。 它们仍然不比随机分类器所能实现的更好：您的模型训练，但是不能泛化。 这是怎么回事？\n",
    "\n",
    "这可能是你能发现的最糟糕的机器学习情况。这表明你的方法存在根本性的问题，而且可能不容易判断是什么。 这里有一些提示。\n",
    "\n",
    "首先，可能是您使用的输入数据根本没有包含足够的信息来预测您的目标：公式化的问题无法解决。 这就是之前我们尝试拟合一个标签被打乱的 MNIST 模型时发生的情况：模型可以训练得很好，但验证准确率会停留在 10%，因为显然不可能用这样的数据集进行泛化。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "proprietary-drinking",
   "metadata": {},
   "source": [
    "也可能是您使用的模型类型不适合手头的问题。 例如，在第 10 章中，您将看到一个时间序列预测问题的示例，其中密集连接的架构无法击败微不足道的基线，而更合适的循环架构确实能够很好地泛化。 使用对问题做出正确假设的模型对于实现泛化至关重要：您应该利用正确的先验架构。\n",
    "\n",
    "在接下来的章节中，您将了解用于各种数据模式（图像、文本、时间序列等）的最佳架构。 一般来说，您应该始终确保阅读有关您正在攻击的任务类型的架构最佳实践，您可能不是第一个尝试它的人。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "outside-strain",
   "metadata": {},
   "source": [
    "### 增加模型容量"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stopped-tumor",
   "metadata": {},
   "source": [
    "如果你设法得到一个合适的模型，验证指标正在下降，而且这似乎至少达到了一定程度的泛化能力，恭喜：你就快到了。 接下来，您需要让您的模型开始过度拟合。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "going-giant",
   "metadata": {},
   "source": [
    "考虑以下在 MNIST 像素上训练的小模型——一个简单的逻辑回归。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "developed-charge",
   "metadata": {},
   "source": [
    "> 清单 5.9 MNIST 上的简单逻辑回归"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "wicked-georgia",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.6716 - accuracy: 0.8339 - val_loss: 0.3588 - val_accuracy: 0.9040\n",
      "Epoch 2/20\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.3508 - accuracy: 0.9038 - val_loss: 0.3078 - val_accuracy: 0.9149\n",
      "Epoch 3/20\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.3156 - accuracy: 0.9127 - val_loss: 0.2893 - val_accuracy: 0.9186\n",
      "Epoch 4/20\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.2991 - accuracy: 0.9169 - val_loss: 0.2835 - val_accuracy: 0.9194\n",
      "Epoch 5/20\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.2897 - accuracy: 0.9190 - val_loss: 0.2772 - val_accuracy: 0.9212\n",
      "Epoch 6/20\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.2828 - accuracy: 0.9212 - val_loss: 0.2730 - val_accuracy: 0.9245\n",
      "Epoch 7/20\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.2781 - accuracy: 0.9229 - val_loss: 0.2698 - val_accuracy: 0.9268\n",
      "Epoch 8/20\n",
      "375/375 [==============================] - 4s 10ms/step - loss: 0.2741 - accuracy: 0.9236 - val_loss: 0.2689 - val_accuracy: 0.9253\n",
      "Epoch 9/20\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.2713 - accuracy: 0.9247 - val_loss: 0.2653 - val_accuracy: 0.9275\n",
      "Epoch 10/20\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.2685 - accuracy: 0.9257 - val_loss: 0.2640 - val_accuracy: 0.9287\n",
      "Epoch 11/20\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.2665 - accuracy: 0.9264 - val_loss: 0.2646 - val_accuracy: 0.9290\n",
      "Epoch 12/20\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.2648 - accuracy: 0.9276 - val_loss: 0.2646 - val_accuracy: 0.9287\n",
      "Epoch 13/20\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.2631 - accuracy: 0.9276 - val_loss: 0.2652 - val_accuracy: 0.9286\n",
      "Epoch 14/20\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.2618 - accuracy: 0.9283 - val_loss: 0.2631 - val_accuracy: 0.9300\n",
      "Epoch 15/20\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.2605 - accuracy: 0.9283 - val_loss: 0.2635 - val_accuracy: 0.9293\n",
      "Epoch 16/20\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.2591 - accuracy: 0.9288 - val_loss: 0.2619 - val_accuracy: 0.9307\n",
      "Epoch 17/20\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.2584 - accuracy: 0.9291 - val_loss: 0.2628 - val_accuracy: 0.9293\n",
      "Epoch 18/20\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.2570 - accuracy: 0.9296 - val_loss: 0.2613 - val_accuracy: 0.9310\n",
      "Epoch 19/20\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.2564 - accuracy: 0.9300 - val_loss: 0.2613 - val_accuracy: 0.9312\n",
      "Epoch 20/20\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.2554 - accuracy: 0.9304 - val_loss: 0.2628 - val_accuracy: 0.9295\n",
      "time: 57.8 s (started: 2021-07-25 20:51:17 +08:00)\n"
     ]
    }
   ],
   "source": [
    "model = keras.Sequential([layers.Dense(10, activation=\"softmax\")])\n",
    "\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"sparse_categorical_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "history_small_model = model.fit(\n",
    "    train_images, \n",
    "    train_labels,\n",
    "    epochs=20,\n",
    "    batch_size=128,\n",
    "    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "thrown-audit",
   "metadata": {},
   "source": [
    "您会得到如下所示的损失曲线："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "hired-essay",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f89706ba5b0>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg height=\"277.314375pt\" version=\"1.1\" viewBox=\"0 0 392.14375 277.314375\" width=\"392.14375pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2021-07-25T20:52:17.802409</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.4.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 277.314375 \n",
       "L 392.14375 277.314375 \n",
       "L 392.14375 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 50.14375 239.758125 \n",
       "L 384.94375 239.758125 \n",
       "L 384.94375 22.318125 \n",
       "L 50.14375 22.318125 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"md063600a93\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"89.39064\" xlink:href=\"#md063600a93\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 2.5 -->\n",
       "      <g transform=\"translate(81.439077 254.356562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\n",
       "        <path d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" id=\"DejaVuSans-2e\" transform=\"scale(0.015625)\"/>\n",
       "        <path d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" id=\"DejaVuSans-35\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"129.438487\" xlink:href=\"#md063600a93\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 5.0 -->\n",
       "      <g transform=\"translate(121.486924 254.356562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"169.486334\" xlink:href=\"#md063600a93\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 7.5 -->\n",
       "      <g transform=\"translate(161.534771 254.356562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 525 4666 \n",
       "L 3525 4666 \n",
       "L 3525 4397 \n",
       "L 1831 0 \n",
       "L 1172 0 \n",
       "L 2766 4134 \n",
       "L 525 4134 \n",
       "L 525 4666 \n",
       "z\n",
       "\" id=\"DejaVuSans-37\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-37\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"209.534181\" xlink:href=\"#md063600a93\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 10.0 -->\n",
       "      <g transform=\"translate(198.401368 254.356562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"249.582028\" xlink:href=\"#md063600a93\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 12.5 -->\n",
       "      <g transform=\"translate(238.449215 254.356562)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"289.629874\" xlink:href=\"#md063600a93\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 15.0 -->\n",
       "      <g transform=\"translate(278.497062 254.356562)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"329.677721\" xlink:href=\"#md063600a93\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 17.5 -->\n",
       "      <g transform=\"translate(318.544909 254.356562)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-37\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"369.725568\" xlink:href=\"#md063600a93\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 20.0 -->\n",
       "      <g transform=\"translate(358.592756 254.356562)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"127.246094\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_9\">\n",
       "     <!-- Epochs -->\n",
       "     <g transform=\"translate(199.628125 268.034687)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 628 4666 \n",
       "L 3578 4666 \n",
       "L 3578 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2753 \n",
       "L 3481 2753 \n",
       "L 3481 2222 \n",
       "L 1259 2222 \n",
       "L 1259 531 \n",
       "L 3634 531 \n",
       "L 3634 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" id=\"DejaVuSans-45\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" id=\"DejaVuSans-70\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" id=\"DejaVuSans-63\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" id=\"DejaVuSans-68\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" id=\"DejaVuSans-73\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-45\"/>\n",
       "      <use x=\"63.183594\" xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use x=\"126.660156\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"187.841797\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"242.822266\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"306.201172\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"mab01540b17\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#mab01540b17\" y=\"232.445655\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.26 -->\n",
       "      <g transform=\"translate(20.878125 236.244874)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" id=\"DejaVuSans-36\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#mab01540b17\" y=\"191.896373\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.28 -->\n",
       "      <g transform=\"translate(20.878125 195.695592)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 2034 2216 \n",
       "Q 1584 2216 1326 1975 \n",
       "Q 1069 1734 1069 1313 \n",
       "Q 1069 891 1326 650 \n",
       "Q 1584 409 2034 409 \n",
       "Q 2484 409 2743 651 \n",
       "Q 3003 894 3003 1313 \n",
       "Q 3003 1734 2745 1975 \n",
       "Q 2488 2216 2034 2216 \n",
       "z\n",
       "M 1403 2484 \n",
       "Q 997 2584 770 2862 \n",
       "Q 544 3141 544 3541 \n",
       "Q 544 4100 942 4425 \n",
       "Q 1341 4750 2034 4750 \n",
       "Q 2731 4750 3128 4425 \n",
       "Q 3525 4100 3525 3541 \n",
       "Q 3525 3141 3298 2862 \n",
       "Q 3072 2584 2669 2484 \n",
       "Q 3125 2378 3379 2068 \n",
       "Q 3634 1759 3634 1313 \n",
       "Q 3634 634 3220 271 \n",
       "Q 2806 -91 2034 -91 \n",
       "Q 1263 -91 848 271 \n",
       "Q 434 634 434 1313 \n",
       "Q 434 1759 690 2068 \n",
       "Q 947 2378 1403 2484 \n",
       "z\n",
       "M 1172 3481 \n",
       "Q 1172 3119 1398 2916 \n",
       "Q 1625 2713 2034 2713 \n",
       "Q 2441 2713 2670 2916 \n",
       "Q 2900 3119 2900 3481 \n",
       "Q 2900 3844 2670 4047 \n",
       "Q 2441 4250 2034 4250 \n",
       "Q 1625 4250 1398 4047 \n",
       "Q 1172 3844 1172 3481 \n",
       "z\n",
       "\" id=\"DejaVuSans-38\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-38\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#mab01540b17\" y=\"151.347091\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0.30 -->\n",
       "      <g transform=\"translate(20.878125 155.14631)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" id=\"DejaVuSans-33\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#mab01540b17\" y=\"110.797809\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 0.32 -->\n",
       "      <g transform=\"translate(20.878125 114.597028)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#mab01540b17\" y=\"70.248527\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 0.34 -->\n",
       "      <g transform=\"translate(20.878125 74.047746)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#mab01540b17\" y=\"29.699245\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 0.36 -->\n",
       "      <g transform=\"translate(20.878125 33.498464)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_16\">\n",
       "     <!-- Loss -->\n",
       "     <g transform=\"translate(14.798438 142.005312)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" id=\"DejaVuSans-4c\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\"/>\n",
       "      <use x=\"53.962891\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"115.144531\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "      <use x=\"167.244141\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_15\">\n",
       "    <path clip-path=\"url(#pdc06517c22)\" d=\"M 65.361932 32.201761 \n",
       "L 81.381071 135.590706 \n",
       "L 97.400209 173.025076 \n",
       "L 113.419348 184.735141 \n",
       "L 129.438487 197.642611 \n",
       "L 145.457626 206.109343 \n",
       "L 161.476764 212.650933 \n",
       "L 177.495903 214.502177 \n",
       "L 193.515042 221.662017 \n",
       "L 209.534181 224.326678 \n",
       "L 225.553319 223.135738 \n",
       "L 241.572458 223.209514 \n",
       "L 257.591597 221.88522 \n",
       "L 273.610736 226.211759 \n",
       "L 289.629874 225.421424 \n",
       "L 305.649013 228.65086 \n",
       "L 321.668152 226.671821 \n",
       "L 337.687291 229.874489 \n",
       "L 353.706429 229.834126 \n",
       "L 369.725568 226.728679 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 50.14375 239.758125 \n",
       "L 50.14375 22.318125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 384.94375 239.758125 \n",
       "L 384.94375 22.318125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 50.14375 239.758125 \n",
       "L 384.94375 239.758125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 50.14375 22.318125 \n",
       "L 384.94375 22.318125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"text_17\">\n",
       "    <!-- Effect of insufficient model capacity on validation loss -->\n",
       "    <g transform=\"translate(55.33375 16.318125)scale(0.12 -0.12)\">\n",
       "     <defs>\n",
       "      <path d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" id=\"DejaVuSans-66\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" id=\"DejaVuSans-65\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" id=\"DejaVuSans-74\" transform=\"scale(0.015625)\"/>\n",
       "      <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" id=\"DejaVuSans-69\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" id=\"DejaVuSans-6e\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" id=\"DejaVuSans-75\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" id=\"DejaVuSans-6d\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" id=\"DejaVuSans-64\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" id=\"DejaVuSans-6c\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" id=\"DejaVuSans-79\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 563 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2284 0 \n",
       "L 1503 0 \n",
       "L 191 3500 \n",
       "z\n",
       "\" id=\"DejaVuSans-76\" transform=\"scale(0.015625)\"/>\n",
       "     </defs>\n",
       "     <use xlink:href=\"#DejaVuSans-45\"/>\n",
       "     <use x=\"63.183594\" xlink:href=\"#DejaVuSans-66\"/>\n",
       "     <use x=\"98.388672\" xlink:href=\"#DejaVuSans-66\"/>\n",
       "     <use x=\"133.59375\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "     <use x=\"195.117188\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "     <use x=\"250.097656\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "     <use x=\"289.306641\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"321.09375\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "     <use x=\"382.275391\" xlink:href=\"#DejaVuSans-66\"/>\n",
       "     <use x=\"417.480469\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"449.267578\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"477.050781\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "     <use x=\"540.429688\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     <use x=\"592.529297\" xlink:href=\"#DejaVuSans-75\"/>\n",
       "     <use x=\"655.908203\" xlink:href=\"#DejaVuSans-66\"/>\n",
       "     <use x=\"691.113281\" xlink:href=\"#DejaVuSans-66\"/>\n",
       "     <use x=\"726.318359\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"754.101562\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "     <use x=\"809.082031\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"836.865234\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "     <use x=\"898.388672\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "     <use x=\"961.767578\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "     <use x=\"1000.976562\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"1032.763672\" xlink:href=\"#DejaVuSans-6d\"/>\n",
       "     <use x=\"1130.175781\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "     <use x=\"1191.357422\" xlink:href=\"#DejaVuSans-64\"/>\n",
       "     <use x=\"1254.833984\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "     <use x=\"1316.357422\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "     <use x=\"1344.140625\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"1375.927734\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "     <use x=\"1430.908203\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "     <use x=\"1492.1875\" xlink:href=\"#DejaVuSans-70\"/>\n",
       "     <use x=\"1555.664062\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "     <use x=\"1616.943359\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "     <use x=\"1671.923828\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"1699.707031\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "     <use x=\"1738.916016\" xlink:href=\"#DejaVuSans-79\"/>\n",
       "     <use x=\"1798.095703\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"1829.882812\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "     <use x=\"1891.064453\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "     <use x=\"1954.443359\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"1986.230469\" xlink:href=\"#DejaVuSans-76\"/>\n",
       "     <use x=\"2045.410156\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "     <use x=\"2106.689453\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "     <use x=\"2134.472656\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"2162.255859\" xlink:href=\"#DejaVuSans-64\"/>\n",
       "     <use x=\"2225.732422\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "     <use x=\"2287.011719\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "     <use x=\"2326.220703\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"2354.003906\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "     <use x=\"2415.185547\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "     <use x=\"2478.564453\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"2510.351562\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "     <use x=\"2538.134766\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "     <use x=\"2599.316406\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     <use x=\"2651.416016\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 274.06875 44.99625 \n",
       "L 377.94375 44.99625 \n",
       "Q 379.94375 44.99625 379.94375 42.99625 \n",
       "L 379.94375 29.318125 \n",
       "Q 379.94375 27.318125 377.94375 27.318125 \n",
       "L 274.06875 27.318125 \n",
       "Q 272.06875 27.318125 272.06875 29.318125 \n",
       "L 272.06875 42.99625 \n",
       "Q 272.06875 44.99625 274.06875 44.99625 \n",
       "z\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_16\">\n",
       "     <path d=\"M 276.06875 35.416562 \n",
       "L 296.06875 35.416562 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_17\"/>\n",
       "    <g id=\"text_18\">\n",
       "     <!-- Validation loss -->\n",
       "     <g transform=\"translate(304.06875 38.916562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 1831 0 \n",
       "L 50 4666 \n",
       "L 709 4666 \n",
       "L 2188 738 \n",
       "L 3669 4666 \n",
       "L 4325 4666 \n",
       "L 2547 0 \n",
       "L 1831 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-56\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-56\"/>\n",
       "      <use x=\"60.658203\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"121.9375\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use x=\"149.720703\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"177.503906\" xlink:href=\"#DejaVuSans-64\"/>\n",
       "      <use x=\"240.980469\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"302.259766\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use x=\"341.46875\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"369.251953\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"430.433594\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use x=\"493.8125\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"525.599609\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use x=\"553.382812\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"614.564453\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "      <use x=\"666.664062\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pdc06517c22\">\n",
       "   <rect height=\"217.44\" width=\"334.8\" x=\"50.14375\" y=\"22.318125\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 277 ms (started: 2021-07-25 20:52:17 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "val_loss = history_small_model.history[\"val_loss\"]\n",
    "\n",
    "epochs = range(1, 21)\n",
    "plt.plot(epochs, val_loss, \"b--\",label=\"Validation loss\")\n",
    "\n",
    "plt.title(\"Effect of insufficient model capacity on validation loss\")\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "binary-state",
   "metadata": {},
   "source": [
    "验证指标似乎停滞不前，或者改善非常缓慢，而不是达到峰值和逆转过程。 验证损失达到 0.26 并保持在那里。 你可以拟合，但你不能明显地过度拟合，即使在对训练数据进行多次迭代之后也是如此。 在您的职业生涯中，您可能会经常遇到类似的曲线。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "musical-intellectual",
   "metadata": {},
   "source": [
    "请记住，过拟合应该总是可能的。 就像“训练损失不会下降”的问题一样，这是一个总是可以解决的问题。 如果您似乎无法过度拟合，则可能是模型的表示能力存在问题：您将需要一个更大的模型。 一个多，也就是说，能够存储更多的信息。 您可以通过添加更多层、使用更大的层（具有更多参数的层）或使用更适合手头问题的层（更好的架构先验）来提高表示能力。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "honest-howard",
   "metadata": {},
   "source": [
    "让我们尝试训练一个更大的模型，一个有两个中间层，每个层有 96 个单元："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "computational-alabama",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "375/375 [==============================] - 4s 8ms/step - loss: 0.8065 - accuracy: 0.7436 - val_loss: 0.3944 - val_accuracy: 0.8880\n",
      "Epoch 2/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.3578 - accuracy: 0.8922 - val_loss: 0.2840 - val_accuracy: 0.9124\n",
      "Epoch 3/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.2745 - accuracy: 0.9172 - val_loss: 0.2243 - val_accuracy: 0.9338\n",
      "Epoch 4/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.2149 - accuracy: 0.9348 - val_loss: 0.1762 - val_accuracy: 0.9480\n",
      "Epoch 5/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.1732 - accuracy: 0.9475 - val_loss: 0.1582 - val_accuracy: 0.9516\n",
      "Epoch 6/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.1422 - accuracy: 0.9562 - val_loss: 0.1456 - val_accuracy: 0.9557\n",
      "Epoch 7/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.1181 - accuracy: 0.9641 - val_loss: 0.1262 - val_accuracy: 0.9623\n",
      "Epoch 8/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.1014 - accuracy: 0.9691 - val_loss: 0.1069 - val_accuracy: 0.9688\n",
      "Epoch 9/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0881 - accuracy: 0.9732 - val_loss: 0.1124 - val_accuracy: 0.9666\n",
      "Epoch 10/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0768 - accuracy: 0.9770 - val_loss: 0.0945 - val_accuracy: 0.9722\n",
      "Epoch 11/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0677 - accuracy: 0.9792 - val_loss: 0.1133 - val_accuracy: 0.9683\n",
      "Epoch 12/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0605 - accuracy: 0.9815 - val_loss: 0.0948 - val_accuracy: 0.9744\n",
      "Epoch 13/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0533 - accuracy: 0.9834 - val_loss: 0.0931 - val_accuracy: 0.9744\n",
      "Epoch 14/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0477 - accuracy: 0.9850 - val_loss: 0.1002 - val_accuracy: 0.9725\n",
      "Epoch 15/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0442 - accuracy: 0.9862 - val_loss: 0.0829 - val_accuracy: 0.9784\n",
      "Epoch 16/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0383 - accuracy: 0.9879 - val_loss: 0.1075 - val_accuracy: 0.9696\n",
      "Epoch 17/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0349 - accuracy: 0.9885 - val_loss: 0.0907 - val_accuracy: 0.9758\n",
      "Epoch 18/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0324 - accuracy: 0.9899 - val_loss: 0.1117 - val_accuracy: 0.9703\n",
      "Epoch 19/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0284 - accuracy: 0.9909 - val_loss: 0.0868 - val_accuracy: 0.9788\n",
      "Epoch 20/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0256 - accuracy: 0.9917 - val_loss: 0.0875 - val_accuracy: 0.9785\n",
      "Epoch 21/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0229 - accuracy: 0.9924 - val_loss: 0.0927 - val_accuracy: 0.9781\n",
      "Epoch 22/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0209 - accuracy: 0.9936 - val_loss: 0.0888 - val_accuracy: 0.9789\n",
      "Epoch 23/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0190 - accuracy: 0.9937 - val_loss: 0.0908 - val_accuracy: 0.9796\n",
      "Epoch 24/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0173 - accuracy: 0.9942 - val_loss: 0.0992 - val_accuracy: 0.9783\n",
      "Epoch 25/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0161 - accuracy: 0.9949 - val_loss: 0.0985 - val_accuracy: 0.9789\n",
      "Epoch 26/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0150 - accuracy: 0.9952 - val_loss: 0.1041 - val_accuracy: 0.9763\n",
      "Epoch 27/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0132 - accuracy: 0.9957 - val_loss: 0.1075 - val_accuracy: 0.9784\n",
      "Epoch 28/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0120 - accuracy: 0.9963 - val_loss: 0.1101 - val_accuracy: 0.9779\n",
      "Epoch 29/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0112 - accuracy: 0.9961 - val_loss: 0.1054 - val_accuracy: 0.9787\n",
      "Epoch 30/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0107 - accuracy: 0.9966 - val_loss: 0.1273 - val_accuracy: 0.9757\n",
      "Epoch 31/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0093 - accuracy: 0.9970 - val_loss: 0.1055 - val_accuracy: 0.9808\n",
      "Epoch 32/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0083 - accuracy: 0.9973 - val_loss: 0.1099 - val_accuracy: 0.9797\n",
      "Epoch 33/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0081 - accuracy: 0.9973 - val_loss: 0.1100 - val_accuracy: 0.9800\n",
      "Epoch 34/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0082 - accuracy: 0.9973 - val_loss: 0.1212 - val_accuracy: 0.9761\n",
      "Epoch 35/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0073 - accuracy: 0.9977 - val_loss: 0.1303 - val_accuracy: 0.9762\n",
      "Epoch 36/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0064 - accuracy: 0.9978 - val_loss: 0.1157 - val_accuracy: 0.9810\n",
      "Epoch 37/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0074 - accuracy: 0.9977 - val_loss: 0.1433 - val_accuracy: 0.9755\n",
      "Epoch 38/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0065 - accuracy: 0.9978 - val_loss: 0.1232 - val_accuracy: 0.9804\n",
      "Epoch 39/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0061 - accuracy: 0.9978 - val_loss: 0.1296 - val_accuracy: 0.9788\n",
      "Epoch 40/50\n",
      "375/375 [==============================] - 4s 10ms/step - loss: 0.0054 - accuracy: 0.9981 - val_loss: 0.1254 - val_accuracy: 0.9808\n",
      "Epoch 41/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0044 - accuracy: 0.9984 - val_loss: 0.1540 - val_accuracy: 0.9747\n",
      "Epoch 42/50\n",
      "375/375 [==============================] - 3s 9ms/step - loss: 0.0046 - accuracy: 0.9984 - val_loss: 0.1348 - val_accuracy: 0.9795\n",
      "Epoch 43/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0050 - accuracy: 0.9985 - val_loss: 0.1438 - val_accuracy: 0.9787\n",
      "Epoch 44/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0053 - accuracy: 0.9983 - val_loss: 0.1418 - val_accuracy: 0.9792\n",
      "Epoch 45/50\n",
      "375/375 [==============================] - 3s 8ms/step - loss: 0.0040 - accuracy: 0.9986 - val_loss: 0.1475 - val_accuracy: 0.9790\n",
      "Epoch 46/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0046 - accuracy: 0.9984 - val_loss: 0.1398 - val_accuracy: 0.9808\n",
      "Epoch 47/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0042 - accuracy: 0.9986 - val_loss: 0.1647 - val_accuracy: 0.9764\n",
      "Epoch 48/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0037 - accuracy: 0.9990 - val_loss: 0.1418 - val_accuracy: 0.9805\n",
      "Epoch 49/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0043 - accuracy: 0.9985 - val_loss: 0.1567 - val_accuracy: 0.9777\n",
      "Epoch 50/50\n",
      "375/375 [==============================] - 3s 7ms/step - loss: 0.0036 - accuracy: 0.9990 - val_loss: 0.1465 - val_accuracy: 0.9808\n",
      "time: 2min 25s (started: 2021-07-25 21:04:32 +08:00)\n"
     ]
    }
   ],
   "source": [
    "model = keras.Sequential([\n",
    "    layers.Dense(1024, activation=\"relu\"),\n",
    "    layers.Dense(1024, activation=\"relu\"),\n",
    "    layers.Dense(10, activation=\"softmax\"),\n",
    "])\n",
    "\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"sparse_categorical_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "history_large_model = model.fit(\n",
    "    train_images, \n",
    "    train_labels,\n",
    "    epochs=50,\n",
    "    batch_size=128,\n",
    "    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "enormous-scoop",
   "metadata": {},
   "source": [
    "训练曲线现在看起来和它们应该的完全一样：模型拟合得很快，并且在 8 个 epoch 后开始过度拟合。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "tested-finger",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f88fa875e80>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg height=\"277.314375pt\" version=\"1.1\" viewBox=\"0 0 392.14375 277.314375\" width=\"392.14375pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2021-07-25T21:08:58.442982</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.4.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 277.314375 \n",
       "L 392.14375 277.314375 \n",
       "L 392.14375 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 50.14375 239.758125 \n",
       "L 384.94375 239.758125 \n",
       "L 384.94375 22.318125 \n",
       "L 50.14375 22.318125 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"m4a99c00175\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"59.150429\" xlink:href=\"#m4a99c00175\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(55.969179 254.356562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"121.265457\" xlink:href=\"#m4a99c00175\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(114.902957 254.356562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"183.380485\" xlink:href=\"#m4a99c00175\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(177.017985 254.356562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" id=\"DejaVuSans-32\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"245.495513\" xlink:href=\"#m4a99c00175\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 30 -->\n",
       "      <g transform=\"translate(239.133013 254.356562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" id=\"DejaVuSans-33\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"307.61054\" xlink:href=\"#m4a99c00175\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(301.24804 254.356562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" id=\"DejaVuSans-34\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"369.725568\" xlink:href=\"#m4a99c00175\" y=\"239.758125\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 50 -->\n",
       "      <g transform=\"translate(363.363068 254.356562)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 691 4666 \n",
       "L 3169 4666 \n",
       "L 3169 4134 \n",
       "L 1269 4134 \n",
       "L 1269 2991 \n",
       "Q 1406 3038 1543 3061 \n",
       "Q 1681 3084 1819 3084 \n",
       "Q 2600 3084 3056 2656 \n",
       "Q 3513 2228 3513 1497 \n",
       "Q 3513 744 3044 326 \n",
       "Q 2575 -91 1722 -91 \n",
       "Q 1428 -91 1123 -41 \n",
       "Q 819 9 494 109 \n",
       "L 494 744 \n",
       "Q 775 591 1075 516 \n",
       "Q 1375 441 1709 441 \n",
       "Q 2250 441 2565 725 \n",
       "Q 2881 1009 2881 1497 \n",
       "Q 2881 1984 2565 2268 \n",
       "Q 2250 2553 1709 2553 \n",
       "Q 1456 2553 1204 2497 \n",
       "Q 953 2441 691 2322 \n",
       "L 691 4666 \n",
       "z\n",
       "\" id=\"DejaVuSans-35\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-35\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_7\">\n",
       "     <!-- Epochs -->\n",
       "     <g transform=\"translate(199.628125 268.034687)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 628 4666 \n",
       "L 3578 4666 \n",
       "L 3578 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2753 \n",
       "L 3481 2753 \n",
       "L 3481 2222 \n",
       "L 1259 2222 \n",
       "L 1259 531 \n",
       "L 3634 531 \n",
       "L 3634 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" id=\"DejaVuSans-45\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" id=\"DejaVuSans-70\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" id=\"DejaVuSans-63\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" id=\"DejaVuSans-68\" transform=\"scale(0.015625)\"/>\n",
       "       <path d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" id=\"DejaVuSans-73\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-45\"/>\n",
       "      <use x=\"63.183594\" xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use x=\"126.660156\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"187.841797\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"242.822266\" xlink:href=\"#DejaVuSans-68\"/>\n",
       "      <use x=\"306.201172\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"ma20860eb51\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#ma20860eb51\" y=\"218.994123\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.10 -->\n",
       "      <g transform=\"translate(20.878125 222.793342)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" id=\"DejaVuSans-2e\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#ma20860eb51\" y=\"187.271036\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.15 -->\n",
       "      <g transform=\"translate(20.878125 191.070255)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#ma20860eb51\" y=\"155.54795\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.20 -->\n",
       "      <g transform=\"translate(20.878125 159.347168)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#ma20860eb51\" y=\"123.824863\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.25 -->\n",
       "      <g transform=\"translate(20.878125 127.624081)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#ma20860eb51\" y=\"92.101776\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0.30 -->\n",
       "      <g transform=\"translate(20.878125 95.900994)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#ma20860eb51\" y=\"60.378689\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 0.35 -->\n",
       "      <g transform=\"translate(20.878125 64.177908)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"50.14375\" xlink:href=\"#ma20860eb51\" y=\"28.655602\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 0.40 -->\n",
       "      <g transform=\"translate(20.878125 32.454821)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_15\">\n",
       "     <!-- Loss -->\n",
       "     <g transform=\"translate(14.798438 142.005312)rotate(-90)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" id=\"DejaVuSans-4c\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\"/>\n",
       "      <use x=\"53.962891\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"115.144531\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "      <use x=\"167.244141\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_14\">\n",
       "    <path clip-path=\"url(#p234f5ae43b)\" d=\"M 65.361932 32.201761 \n",
       "L 71.573435 102.223018 \n",
       "L 77.784937 140.129208 \n",
       "L 83.99644 170.647948 \n",
       "L 90.207943 182.053241 \n",
       "L 96.419446 190.087349 \n",
       "L 102.630949 202.369123 \n",
       "L 108.842451 214.63899 \n",
       "L 115.053954 211.104857 \n",
       "L 121.265457 222.514367 \n",
       "L 127.47696 210.586903 \n",
       "L 133.688462 222.324814 \n",
       "L 139.899965 223.385209 \n",
       "L 146.111468 218.887928 \n",
       "L 152.322971 229.874489 \n",
       "L 158.534474 214.222901 \n",
       "L 164.745976 224.916645 \n",
       "L 170.957479 211.566181 \n",
       "L 177.168982 227.342347 \n",
       "L 183.380485 226.905449 \n",
       "L 189.591987 223.625587 \n",
       "L 195.80349 226.071005 \n",
       "L 202.014993 224.849185 \n",
       "L 208.226496 219.520746 \n",
       "L 214.437999 219.928421 \n",
       "L 220.649501 216.370549 \n",
       "L 226.861004 214.264802 \n",
       "L 233.072507 212.584707 \n",
       "L 239.28401 215.568935 \n",
       "L 245.495513 201.700389 \n",
       "L 251.707015 215.47532 \n",
       "L 257.918518 212.688926 \n",
       "L 264.130021 212.640128 \n",
       "L 270.341524 205.554046 \n",
       "L 276.553026 199.772466 \n",
       "L 282.764529 209.032219 \n",
       "L 288.976032 191.502069 \n",
       "L 295.187535 204.279816 \n",
       "L 301.399038 200.208703 \n",
       "L 307.61054 202.864628 \n",
       "L 313.822043 184.753545 \n",
       "L 320.033546 196.885158 \n",
       "L 326.245049 191.219492 \n",
       "L 332.456551 192.449211 \n",
       "L 338.668054 188.863718 \n",
       "L 344.879557 193.715301 \n",
       "L 351.09106 177.929586 \n",
       "L 357.302563 192.493551 \n",
       "L 363.514065 182.99923 \n",
       "L 369.725568 189.522706 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 50.14375 239.758125 \n",
       "L 50.14375 22.318125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 384.94375 239.758125 \n",
       "L 384.94375 22.318125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 50.14375 239.758125 \n",
       "L 384.94375 239.758125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 50.14375 22.318125 \n",
       "L 384.94375 22.318125 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"text_16\">\n",
       "    <!-- Effect of insufficient model capacity on validation loss -->\n",
       "    <g transform=\"translate(55.33375 16.318125)scale(0.12 -0.12)\">\n",
       "     <defs>\n",
       "      <path d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" id=\"DejaVuSans-66\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" id=\"DejaVuSans-65\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" id=\"DejaVuSans-74\" transform=\"scale(0.015625)\"/>\n",
       "      <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" id=\"DejaVuSans-69\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" id=\"DejaVuSans-6e\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" id=\"DejaVuSans-75\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" id=\"DejaVuSans-6d\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" id=\"DejaVuSans-64\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" id=\"DejaVuSans-6c\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" id=\"DejaVuSans-79\" transform=\"scale(0.015625)\"/>\n",
       "      <path d=\"M 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 563 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2284 0 \n",
       "L 1503 0 \n",
       "L 191 3500 \n",
       "z\n",
       "\" id=\"DejaVuSans-76\" transform=\"scale(0.015625)\"/>\n",
       "     </defs>\n",
       "     <use xlink:href=\"#DejaVuSans-45\"/>\n",
       "     <use x=\"63.183594\" xlink:href=\"#DejaVuSans-66\"/>\n",
       "     <use x=\"98.388672\" xlink:href=\"#DejaVuSans-66\"/>\n",
       "     <use x=\"133.59375\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "     <use x=\"195.117188\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "     <use x=\"250.097656\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "     <use x=\"289.306641\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"321.09375\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "     <use x=\"382.275391\" xlink:href=\"#DejaVuSans-66\"/>\n",
       "     <use x=\"417.480469\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"449.267578\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"477.050781\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "     <use x=\"540.429688\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     <use x=\"592.529297\" xlink:href=\"#DejaVuSans-75\"/>\n",
       "     <use x=\"655.908203\" xlink:href=\"#DejaVuSans-66\"/>\n",
       "     <use x=\"691.113281\" xlink:href=\"#DejaVuSans-66\"/>\n",
       "     <use x=\"726.318359\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"754.101562\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "     <use x=\"809.082031\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"836.865234\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "     <use x=\"898.388672\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "     <use x=\"961.767578\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "     <use x=\"1000.976562\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"1032.763672\" xlink:href=\"#DejaVuSans-6d\"/>\n",
       "     <use x=\"1130.175781\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "     <use x=\"1191.357422\" xlink:href=\"#DejaVuSans-64\"/>\n",
       "     <use x=\"1254.833984\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "     <use x=\"1316.357422\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "     <use x=\"1344.140625\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"1375.927734\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "     <use x=\"1430.908203\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "     <use x=\"1492.1875\" xlink:href=\"#DejaVuSans-70\"/>\n",
       "     <use x=\"1555.664062\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "     <use x=\"1616.943359\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "     <use x=\"1671.923828\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"1699.707031\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "     <use x=\"1738.916016\" xlink:href=\"#DejaVuSans-79\"/>\n",
       "     <use x=\"1798.095703\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"1829.882812\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "     <use x=\"1891.064453\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "     <use x=\"1954.443359\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"1986.230469\" xlink:href=\"#DejaVuSans-76\"/>\n",
       "     <use x=\"2045.410156\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "     <use x=\"2106.689453\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "     <use x=\"2134.472656\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"2162.255859\" xlink:href=\"#DejaVuSans-64\"/>\n",
       "     <use x=\"2225.732422\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "     <use x=\"2287.011719\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "     <use x=\"2326.220703\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "     <use x=\"2354.003906\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "     <use x=\"2415.185547\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "     <use x=\"2478.564453\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "     <use x=\"2510.351562\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "     <use x=\"2538.134766\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "     <use x=\"2599.316406\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     <use x=\"2651.416016\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 274.06875 44.99625 \n",
       "L 377.94375 44.99625 \n",
       "Q 379.94375 44.99625 379.94375 42.99625 \n",
       "L 379.94375 29.318125 \n",
       "Q 379.94375 27.318125 377.94375 27.318125 \n",
       "L 274.06875 27.318125 \n",
       "Q 272.06875 27.318125 272.06875 29.318125 \n",
       "L 272.06875 42.99625 \n",
       "Q 272.06875 44.99625 274.06875 44.99625 \n",
       "z\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_15\">\n",
       "     <path d=\"M 276.06875 35.416562 \n",
       "L 296.06875 35.416562 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_16\"/>\n",
       "    <g id=\"text_17\">\n",
       "     <!-- Validation loss -->\n",
       "     <g transform=\"translate(304.06875 38.916562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 1831 0 \n",
       "L 50 4666 \n",
       "L 709 4666 \n",
       "L 2188 738 \n",
       "L 3669 4666 \n",
       "L 4325 4666 \n",
       "L 2547 0 \n",
       "L 1831 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-56\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-56\"/>\n",
       "      <use x=\"60.658203\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"121.9375\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use x=\"149.720703\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"177.503906\" xlink:href=\"#DejaVuSans-64\"/>\n",
       "      <use x=\"240.980469\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"302.259766\" xlink:href=\"#DejaVuSans-74\"/>\n",
       "      <use x=\"341.46875\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"369.251953\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"430.433594\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use x=\"493.8125\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"525.599609\" xlink:href=\"#DejaVuSans-6c\"/>\n",
       "      <use x=\"553.382812\" xlink:href=\"#DejaVuSans-6f\"/>\n",
       "      <use x=\"614.564453\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "      <use x=\"666.664062\" xlink:href=\"#DejaVuSans-73\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p234f5ae43b\">\n",
       "   <rect height=\"217.44\" width=\"334.8\" x=\"50.14375\" y=\"22.318125\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 329 ms (started: 2021-07-25 21:08:58 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "val_loss = history_large_model.history[\"val_loss\"]\n",
    "\n",
    "epochs = range(1, 51)\n",
    "plt.plot(epochs, val_loss, \"b--\",label=\"Validation loss\")\n",
    "\n",
    "plt.title(\"Effect of insufficient model capacity on validation loss\")\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "generic-hollywood",
   "metadata": {},
   "source": [
    "训练曲线现在看起来和它们应该的完全一样：模型拟合得很快，并且在 20 个 epoch 后开始过度拟合。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cooperative-original",
   "metadata": {},
   "source": [
    "## 提高泛化能力"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "horizontal-magazine",
   "metadata": {},
   "source": [
    "一旦您的模型显示出一定的泛化能力并能够过拟合，就该将注意力转向最大化泛化。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "owned-jersey",
   "metadata": {},
   "source": [
    "### 数据集管理"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "posted-retro",
   "metadata": {},
   "source": [
    "您已经了解到深度学习中的泛化源于数据的潜在结构。 如果您的数据可以在样本之间平滑插入，那么您将能够训练一个泛化的深度学习模型。 如果您的问题过于嘈杂或从根本上是离散的，例如列表排序，深度学习将无济于事。 深度学习是曲线拟合，而不是魔法。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "supreme-documentary",
   "metadata": {},
   "source": [
    "因此，确保您使用的是合适的数据集至关重要。 在数据收集上花费更多的精力和金钱几乎总是比在开发更好的模型上花费相同的钱获得更高的投资回报。\n",
    "* 确保你有足够的数据。 请记住，您需要对输入交叉输出空间进行密集采样。 更多的数据将产生更好的模型。 有时，一开始看起来不可能的问题可以通过更大的数据集解决。\n",
    "* 最大限度地减少标签错误——可视化您的输入以检查异常，并校对您的标签。\n",
    "* 清理您的数据并处理缺失值（我们将在下一章中介绍）。\n",
    "* 如果您有很多功能，但不确定哪些功能真正有用，请进行功能选择。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "durable-azerbaijan",
   "metadata": {},
   "source": [
    "可以提高数据泛化潜力的一个特别重要的方法是特征工程。 对于大多数机器学习问题，特征工程是成功的关键因素。 让我们来看看。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "interested-definition",
   "metadata": {},
   "source": [
    "### 特征工程"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "spanish-vegetarian",
   "metadata": {},
   "source": [
    "特征工程是利用您自己对数据和手头机器学习算法（在本例中为神经网络）的知识，通过对数据应用硬编码（非学习）转换来使算法更好地工作的过程 它进入模型。 在许多情况下，期望机器学习模型能够从完全任意的数据中学习是不合理的。 数据需要以某种方式呈现给模型\n",
    "使模型的工作更轻松。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aboriginal-ratio",
   "metadata": {},
   "source": [
    "让我们看一个直观的例子。 假设您正在尝试开发一个模型，该模型可以将时钟图像作为输入并输出一天中的时间（见图 5.16）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "several-volunteer",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gstjcprvryj30ro0pmq4v.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cathedral-tourism",
   "metadata": {},
   "source": [
    "如果您选择使用图像的原始像素作为输入数据，那么您将面临一个棘手的机器学习问题。 你需要一个卷积神经网络来解决它，你必须花费相当多的计算资源来训练网络。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "western-western",
   "metadata": {},
   "source": [
    "但是如果你已经从高层次理解了这个问题（你理解人类如何在钟面上读时间），那么你可以为机器学习想出更好的输入特征\n",
    "算法：例如，很容易写一个五行 Python 脚本来跟踪时钟指针的黑色像素并输出每根指针的尖端坐标。 然后一个简单的 (x, y) 机器学习算法可以学习将这些坐标与一天中的适当时间相关联。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "deluxe-audio",
   "metadata": {},
   "source": [
    "您可以更进一步：进行坐标更改，并将坐标表示为相对于图像中心的极坐标 (x, y) 坐标。 您的输入将成为每个时钟指针的 theta 角。 此时，您的功能使问题变得如此简单，以至于不需要机器学习； 一个简单的舍入操作和字典查找就足以恢复一天中的大致时间。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "broke-youth",
   "metadata": {},
   "source": [
    "这就是特征工程的本质：通过以更简单的方式表达问题，从而使问题更容易。 使潜在流形更平滑、更简单、更有条理。 它通常需要深入了解问题。\n",
    "\n",
    "在深度学习之前，特征工程曾经是机器学习工作流程中最重要的部分，因为经典的浅层算法没有足够丰富的假设空间来自行学习有用的特征。 您将数据呈现给算法的方式对其成功至关重要。 例如，在卷积神经网络在 MNIST 数字分类问题上取得成功之前，解决方案通常基于\n",
    "硬编码特征，例如数字图像中的循环次数、图像中每个数字的高度、像素值的直方图等。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adjacent-apple",
   "metadata": {},
   "source": [
    "幸运的是，现代深度学习消除了对大多数特征工程的需求，因为神经网络能够从原始数据中自动提取有用的特征。 这是否意味着只要使用深度神经网络就不必担心特征工程？ 不，有两个原因：\n",
    "* 好的功能仍然可以让您在使用更少资源的同时更优雅地解决问题。 例如，使用卷积神经网络解决读取钟面的问题是荒谬的。\n",
    "* 好的特性让你用更少的数据解决问题。 深度学习模型自行学习特征的能力依赖于大量可用的训练数据； 如果您只有几个样本，那么其特征中的信息值就变得至关重要。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "limited-charm",
   "metadata": {},
   "source": [
    "### 使用提前停止"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alleged-shore",
   "metadata": {},
   "source": [
    "在深度学习中，我们总是使用过度参数化的模型：它们的自由度比拟合数据潜在流形所需的最小自由度大得多。 这种过度参数化不是问题，因为您永远无法完全拟合深度学习模型。 这样的契合根本无法推广。 在达到最小可能的训练损失之前，您总是会中断训练。\n",
    "\n",
    "在训练期间找到达到最可泛化拟合的确切点（欠拟合曲线和过拟合曲线之间的确切边界）是提高泛化能力的最有效方法之一。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "humanitarian-syria",
   "metadata": {},
   "source": [
    "在前一章的例子中，我们首先训练我们的模型比需要的时间长，以确定产生最佳验证指标的时期数，然后我们将重新训练一个新模型，精确到该时期数。 这是很标准的。 但是，它需要您做多余的工作，这有时会很昂贵。 当然，您可以在每个 epoch 结束时保存您的模型，然后一旦找到最佳 epoch，就可以重用您拥有的最接近的保存模型。 在 Keras 中，通常使用 EarlyStopping 回调来执行此操作，一旦验证指标停止改进，它就会中断训练，同时记住最知名的模型状态。 你将在第 7 章学习使用回调。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "controlled-gazette",
   "metadata": {},
   "source": [
    "### 正则化你的模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "grave-morgan",
   "metadata": {},
   "source": [
    "正则化技术是一组最佳实践，它们会积极阻碍模型完美拟合训练数据的能力，目的是使模型在验证过程中表现更好。 这被称为“正则化”模型，因为它倾向于使模型更简单、更“规则”、曲线更平滑、更“通用”——因此对训练集不那么具体，并且能够通过更接近潜在的 数据的多样性。\n",
    "\n",
    "请记住，“规范化”模型是一个应始终以准确的评估程序为指导的过程。 如果您可以衡量它，您将只能实现泛化。\n",
    "\n",
    "让我们回顾一些最常见的正则化技术，并在实践中应用它们来改进第 4 章中的电影分类模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hourly-savannah",
   "metadata": {},
   "source": [
    "**缩小网络规模**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abroad-expert",
   "metadata": {},
   "source": [
    "您已经了解到太小的模型不会过拟合。减轻过拟合的最简单方法是减小模型的大小（模型中可学习参数的数量，由层数和每层单元数决定）。如果模型的记忆资源有限，它就不能简单地记忆它的训练数据；因此，为了最小化其损失，它将不得不求助于学习压缩表示\n",
    "关于目标的预测能力——正是我们感兴趣的表示类型。同时，请记住，您应该使用具有足够参数且不会欠拟合的模型：您的模型不应该因记忆而饿死资源。在容量过多和容量不足之间存在折衷。\n",
    "\n",
    "不幸的是，没有神奇的公式来确定正确的层数或每层的正确大小。您必须评估一系列不同的架构（当然是在您的验证集上，而不是在您的测试集上），以便为您的数据找到正确的模型大小。找到合适模型大小的一般工作流程是从相对较少的层和参数开始，然后增加层的大小或添加新层，直到您看到验证损失的收益递减。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "outstanding-decline",
   "metadata": {},
   "source": [
    "让我们在电影评论分类模型上试试这个。 这是我们的原始模型："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "solid-folder",
   "metadata": {},
   "source": [
    "> 清单 5.10 原始模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "loving-belle",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<__array_function__ internals>:5: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
      "/public/huangwei/miniconda3/envs/tensorflow/lib/python3.8/site-packages/tensorflow/python/keras/datasets/imdb.py:155: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
      "  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])\n",
      "/public/huangwei/miniconda3/envs/tensorflow/lib/python3.8/site-packages/tensorflow/python/keras/datasets/imdb.py:156: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
      "  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "30/30 [==============================] - 3s 56ms/step - loss: 0.5699 - accuracy: 0.7497 - val_loss: 0.4523 - val_accuracy: 0.8459\n",
      "Epoch 2/20\n",
      "30/30 [==============================] - 1s 29ms/step - loss: 0.3615 - accuracy: 0.8912 - val_loss: 0.3352 - val_accuracy: 0.8808\n",
      "Epoch 3/20\n",
      "30/30 [==============================] - 1s 29ms/step - loss: 0.2595 - accuracy: 0.9205 - val_loss: 0.2924 - val_accuracy: 0.8901\n",
      "Epoch 4/20\n",
      "30/30 [==============================] - 1s 28ms/step - loss: 0.2008 - accuracy: 0.9380 - val_loss: 0.2791 - val_accuracy: 0.8913\n",
      "Epoch 5/20\n",
      "30/30 [==============================] - 1s 32ms/step - loss: 0.1641 - accuracy: 0.9482 - val_loss: 0.2724 - val_accuracy: 0.8929\n",
      "Epoch 6/20\n",
      "30/30 [==============================] - 1s 37ms/step - loss: 0.1366 - accuracy: 0.9586 - val_loss: 0.2811 - val_accuracy: 0.8878\n",
      "Epoch 7/20\n",
      "30/30 [==============================] - 1s 39ms/step - loss: 0.1149 - accuracy: 0.9651 - val_loss: 0.2970 - val_accuracy: 0.8865\n",
      "Epoch 8/20\n",
      "30/30 [==============================] - 1s 37ms/step - loss: 0.0985 - accuracy: 0.9718 - val_loss: 0.3080 - val_accuracy: 0.8851\n",
      "Epoch 9/20\n",
      "30/30 [==============================] - 1s 37ms/step - loss: 0.0800 - accuracy: 0.9783 - val_loss: 0.3309 - val_accuracy: 0.8803\n",
      "Epoch 10/20\n",
      "30/30 [==============================] - 1s 41ms/step - loss: 0.0677 - accuracy: 0.9829 - val_loss: 0.3434 - val_accuracy: 0.8827\n",
      "Epoch 11/20\n",
      "30/30 [==============================] - 1s 37ms/step - loss: 0.0556 - accuracy: 0.9862 - val_loss: 0.3726 - val_accuracy: 0.8787\n",
      "Epoch 12/20\n",
      "30/30 [==============================] - 1s 42ms/step - loss: 0.0443 - accuracy: 0.9891 - val_loss: 0.4093 - val_accuracy: 0.8729\n",
      "Epoch 13/20\n",
      "30/30 [==============================] - 1s 39ms/step - loss: 0.0368 - accuracy: 0.9922 - val_loss: 0.4210 - val_accuracy: 0.8759\n",
      "Epoch 14/20\n",
      "30/30 [==============================] - 1s 35ms/step - loss: 0.0298 - accuracy: 0.9943 - val_loss: 0.4506 - val_accuracy: 0.8758\n",
      "Epoch 15/20\n",
      "30/30 [==============================] - 1s 40ms/step - loss: 0.0235 - accuracy: 0.9959 - val_loss: 0.4763 - val_accuracy: 0.8735\n",
      "Epoch 16/20\n",
      "30/30 [==============================] - 1s 39ms/step - loss: 0.0171 - accuracy: 0.9976 - val_loss: 0.5646 - val_accuracy: 0.8639\n",
      "Epoch 17/20\n",
      "30/30 [==============================] - 1s 41ms/step - loss: 0.0134 - accuracy: 0.9986 - val_loss: 0.5395 - val_accuracy: 0.8698\n",
      "Epoch 18/20\n",
      "30/30 [==============================] - 1s 32ms/step - loss: 0.0109 - accuracy: 0.9986 - val_loss: 0.5763 - val_accuracy: 0.8699\n",
      "Epoch 19/20\n",
      "30/30 [==============================] - 1s 23ms/step - loss: 0.0080 - accuracy: 0.9991 - val_loss: 0.6249 - val_accuracy: 0.8672\n",
      "Epoch 20/20\n",
      "30/30 [==============================] - 1s 27ms/step - loss: 0.0063 - accuracy: 0.9994 - val_loss: 0.6551 - val_accuracy: 0.8676\n",
      "time: 33.9 s (started: 2021-07-25 22:06:21 +08:00)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.datasets import imdb\n",
    "(train_data, train_labels), _ = imdb.load_data(num_words=10000)\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1.\n",
    "    return results\n",
    "\n",
    "train_data = vectorize_sequences(train_data)\n",
    "\n",
    "model = keras.Sequential([\n",
    "    layers.Dense(16, activation=\"relu\"),\n",
    "    layers.Dense(16, activation=\"relu\"),\n",
    "    layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "history_original = model.fit(train_data, \n",
    "                             train_labels,\n",
    "                             epochs=20, \n",
    "                             batch_size=512, \n",
    "                             validation_split=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "contemporary-essex",
   "metadata": {},
   "source": [
    "现在让我们尝试用这个较小的模型替换它。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ordered-margin",
   "metadata": {},
   "source": [
    "> Listing 5.11 容量较低的模型版本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "amazing-longitude",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "30/30 [==============================] - 3s 63ms/step - loss: 0.5981 - accuracy: 0.7682 - val_loss: 0.5294 - val_accuracy: 0.8470\n",
      "Epoch 2/20\n",
      "30/30 [==============================] - 1s 31ms/step - loss: 0.4554 - accuracy: 0.8757 - val_loss: 0.4327 - val_accuracy: 0.8530\n",
      "Epoch 3/20\n",
      "30/30 [==============================] - 1s 30ms/step - loss: 0.3655 - accuracy: 0.9018 - val_loss: 0.3648 - val_accuracy: 0.8812\n",
      "Epoch 4/20\n",
      "30/30 [==============================] - 1s 30ms/step - loss: 0.2975 - accuracy: 0.9181 - val_loss: 0.3279 - val_accuracy: 0.8845\n",
      "Epoch 5/20\n",
      "30/30 [==============================] - 1s 29ms/step - loss: 0.2486 - accuracy: 0.9286 - val_loss: 0.2983 - val_accuracy: 0.8894\n",
      "Epoch 6/20\n",
      "30/30 [==============================] - 1s 27ms/step - loss: 0.2122 - accuracy: 0.9379 - val_loss: 0.2850 - val_accuracy: 0.8922\n",
      "Epoch 7/20\n",
      "30/30 [==============================] - 1s 32ms/step - loss: 0.1839 - accuracy: 0.9464 - val_loss: 0.2910 - val_accuracy: 0.8827\n",
      "Epoch 8/20\n",
      "30/30 [==============================] - 1s 37ms/step - loss: 0.1623 - accuracy: 0.9532 - val_loss: 0.2743 - val_accuracy: 0.8911\n",
      "Epoch 9/20\n",
      "30/30 [==============================] - 1s 34ms/step - loss: 0.1442 - accuracy: 0.9587 - val_loss: 0.2764 - val_accuracy: 0.8894\n",
      "Epoch 10/20\n",
      "30/30 [==============================] - 1s 34ms/step - loss: 0.1285 - accuracy: 0.9635 - val_loss: 0.2862 - val_accuracy: 0.8874\n",
      "Epoch 11/20\n",
      "30/30 [==============================] - 1s 30ms/step - loss: 0.1151 - accuracy: 0.9685 - val_loss: 0.2945 - val_accuracy: 0.8851\n",
      "Epoch 12/20\n",
      "30/30 [==============================] - 1s 31ms/step - loss: 0.1035 - accuracy: 0.9720 - val_loss: 0.2953 - val_accuracy: 0.8862\n",
      "Epoch 13/20\n",
      "30/30 [==============================] - 1s 32ms/step - loss: 0.0939 - accuracy: 0.9749 - val_loss: 0.3077 - val_accuracy: 0.8847\n",
      "Epoch 14/20\n",
      "30/30 [==============================] - 1s 31ms/step - loss: 0.0840 - accuracy: 0.9781 - val_loss: 0.3144 - val_accuracy: 0.8827\n",
      "Epoch 15/20\n",
      "30/30 [==============================] - 1s 35ms/step - loss: 0.0761 - accuracy: 0.9816 - val_loss: 0.3315 - val_accuracy: 0.8818\n",
      "Epoch 16/20\n",
      "30/30 [==============================] - 1s 34ms/step - loss: 0.0680 - accuracy: 0.9855 - val_loss: 0.3420 - val_accuracy: 0.8809\n",
      "Epoch 17/20\n",
      "30/30 [==============================] - 1s 38ms/step - loss: 0.0616 - accuracy: 0.9859 - val_loss: 0.3608 - val_accuracy: 0.8789\n",
      "Epoch 18/20\n",
      "30/30 [==============================] - 1s 31ms/step - loss: 0.0549 - accuracy: 0.9881 - val_loss: 0.3722 - val_accuracy: 0.8784\n",
      "Epoch 19/20\n",
      "30/30 [==============================] - 1s 33ms/step - loss: 0.0491 - accuracy: 0.9887 - val_loss: 0.3863 - val_accuracy: 0.8779\n",
      "Epoch 20/20\n",
      "30/30 [==============================] - 1s 32ms/step - loss: 0.0436 - accuracy: 0.9907 - val_loss: 0.4108 - val_accuracy: 0.8748\n",
      "time: 22.2 s (started: 2021-07-25 22:08:06 +08:00)\n"
     ]
    }
   ],
   "source": [
    "model = keras.Sequential([\n",
    "    layers.Dense(4, activation=\"relu\"),\n",
    "    layers.Dense(4, activation=\"relu\"),\n",
    "    layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "history_smaller_model = model.fit(train_data, \n",
    "                                  train_labels,\n",
    "                                  epochs=20, \n",
    "                                  batch_size=512, \n",
    "                                  validation_split=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "connected-drunk",
   "metadata": {},
   "source": [
    "图 5.17 显示了原始模型和较小模型的验证损失的比较。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "narrative-daughter",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gstjtpuvrwj31a20twwhx.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "binding-rainbow",
   "metadata": {},
   "source": [
    "如您所见，较小的模型开始过拟合的时间晚于参考模型（在 6 个 epoch 之后而不是 4 个 epoch 之后），并且一旦开始过拟合，其性能下降得更慢。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "recreational-spiritual",
   "metadata": {},
   "source": [
    "现在，让我们将一个容量更大的模型添加到我们的基准测试中——远远超出问题所保证的范围。 虽然使用对于他们试图学习的东西明显过度参数化的模型是标准的，但肯定会有太多的记忆容量这样的事情。 如果您的模型立即开始过度拟合，并且其验证损失曲线看起来不稳定、高方差（尽管不稳定的验证指标也可能是使用不可靠验证过程的症状，例如验证拆分），您就会知道您的模型太大了 太小）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "premier-museum",
   "metadata": {},
   "source": [
    "> 清单 5.12 更高容量的模型版本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "fresh-denver",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "30/30 [==============================] - 4s 74ms/step - loss: 0.5607 - accuracy: 0.7417 - val_loss: 0.3201 - val_accuracy: 0.8687\n",
      "Epoch 2/20\n",
      "30/30 [==============================] - 1s 35ms/step - loss: 0.2606 - accuracy: 0.8982 - val_loss: 0.2700 - val_accuracy: 0.8960\n",
      "Epoch 3/20\n",
      "30/30 [==============================] - 1s 38ms/step - loss: 0.1487 - accuracy: 0.9447 - val_loss: 0.2924 - val_accuracy: 0.8907\n",
      "Epoch 4/20\n",
      "30/30 [==============================] - 1s 37ms/step - loss: 0.1129 - accuracy: 0.9595 - val_loss: 0.3202 - val_accuracy: 0.8873\n",
      "Epoch 5/20\n",
      "30/30 [==============================] - 1s 37ms/step - loss: 0.0691 - accuracy: 0.9825 - val_loss: 0.3796 - val_accuracy: 0.8633\n",
      "Epoch 6/20\n",
      "30/30 [==============================] - 1s 37ms/step - loss: 0.0091 - accuracy: 0.9993 - val_loss: 0.4874 - val_accuracy: 0.8882\n",
      "Epoch 7/20\n",
      "30/30 [==============================] - 1s 45ms/step - loss: 0.0012 - accuracy: 0.9998 - val_loss: 0.5994 - val_accuracy: 0.8846\n",
      "Epoch 8/20\n",
      "30/30 [==============================] - 1s 35ms/step - loss: 0.2294 - accuracy: 0.9761 - val_loss: 0.4848 - val_accuracy: 0.8808\n",
      "Epoch 9/20\n",
      "30/30 [==============================] - 1s 28ms/step - loss: 0.0013 - accuracy: 0.9999 - val_loss: 0.5286 - val_accuracy: 0.8848\n",
      "Epoch 10/20\n",
      "30/30 [==============================] - 1s 29ms/step - loss: 4.3454e-04 - accuracy: 0.9999 - val_loss: 0.5998 - val_accuracy: 0.8850\n",
      "Epoch 11/20\n",
      "30/30 [==============================] - 1s 38ms/step - loss: 7.4334e-05 - accuracy: 1.0000 - val_loss: 0.7028 - val_accuracy: 0.8850\n",
      "Epoch 12/20\n",
      "30/30 [==============================] - 1s 42ms/step - loss: 1.8773e-05 - accuracy: 1.0000 - val_loss: 0.8000 - val_accuracy: 0.8845\n",
      "Epoch 13/20\n",
      "30/30 [==============================] - 1s 38ms/step - loss: 4.6761e-06 - accuracy: 1.0000 - val_loss: 0.8948 - val_accuracy: 0.8828\n",
      "Epoch 14/20\n",
      "30/30 [==============================] - 1s 33ms/step - loss: 1.4219e-06 - accuracy: 1.0000 - val_loss: 0.9808 - val_accuracy: 0.8843\n",
      "Epoch 15/20\n",
      "30/30 [==============================] - 1s 31ms/step - loss: 4.6986e-07 - accuracy: 1.0000 - val_loss: 1.0526 - val_accuracy: 0.8843\n",
      "Epoch 16/20\n",
      "30/30 [==============================] - 1s 35ms/step - loss: 1.6649e-07 - accuracy: 1.0000 - val_loss: 1.1217 - val_accuracy: 0.8833\n",
      "Epoch 17/20\n",
      "30/30 [==============================] - 1s 30ms/step - loss: 6.6767e-08 - accuracy: 1.0000 - val_loss: 1.1799 - val_accuracy: 0.8835\n",
      "Epoch 18/20\n",
      "30/30 [==============================] - 1s 30ms/step - loss: 3.3172e-08 - accuracy: 1.0000 - val_loss: 1.2214 - val_accuracy: 0.8838\n",
      "Epoch 19/20\n",
      "30/30 [==============================] - 1s 31ms/step - loss: 2.0887e-08 - accuracy: 1.0000 - val_loss: 1.2430 - val_accuracy: 0.8834\n",
      "Epoch 20/20\n",
      "30/30 [==============================] - 1s 36ms/step - loss: 1.4957e-08 - accuracy: 1.0000 - val_loss: 1.2650 - val_accuracy: 0.8837\n",
      "time: 24.6 s (started: 2021-07-25 22:12:20 +08:00)\n"
     ]
    }
   ],
   "source": [
    "model = keras.Sequential([\n",
    "    layers.Dense(512, activation=\"relu\"),\n",
    "    layers.Dense(512, activation=\"relu\"),\n",
    "    layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "history_larger_model = model.fit(train_data, \n",
    "                                 train_labels,\n",
    "                                 epochs=20, \n",
    "                                 batch_size=512, \n",
    "                                 validation_split=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "corporate-advertising",
   "metadata": {},
   "source": [
    "图 5.18 显示了较大模型与参考模型相比的情况。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "surgical-panama",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gstjxxt1b9j313m0r8419.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "seven-amendment",
   "metadata": {},
   "source": [
    "较大的模型几乎立即开始过拟合，仅在一个 epoch 之后，并且过拟合得更严重。 它的验证损失也更嘈杂。 它很快使训练损失接近于零。 模型容量越大，对训练数据建模的速度就越快（导致训练损失较低），但越容易过拟合（导致训练和验证损失之间存在较大差异）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "every-tooth",
   "metadata": {},
   "source": [
    "**增加权重正则化**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "enclosed-knife",
   "metadata": {},
   "source": [
    "您可能熟悉奥卡姆剃刀原理：对某事给出两种解释，最可能正确的解释是最简单的解释——假设较少的解释。 这个想法也适用于神经网络学习的模型：给定一些训练数据和网络架构，多组权重值（multiple）可以模型解释数据。 与复杂模型相比，简单模型不太可能过度拟合。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "generic-exchange",
   "metadata": {},
   "source": [
    "在这种情况下，一个简单的模型是参数值分布具有较少熵的模型（或具有较少参数的模型，如您在上一节中所见）。 因此，减轻过度拟合的一种常见方法是通过强制其权重仅取较小值来限制模型的复杂性，这使得权重值的分布更加规则。 这称为权重正则化，它是通过向模型的损失函数添加与具有大权重相关的成本来完成的。 这个成本有两种形式：\n",
    "* L1 正则化——增加的成本与权重系数范数（权重的 L1）的绝对值成正比。\n",
    "* L2 正则化——增加的成本与权重系数 L2 范数（权重的值）的平方成正比。 L2 正则化在神经网络的上下文中也称为权重衰减。 不要让不同的名称混淆你：权重衰减在数学上与 L2 正则化相同。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "alleged-makeup",
   "metadata": {},
   "source": [
    "在 Keras 中，通过将权重正则化器实例作为关键字参数传递给层来添加权重正则化。 让我们将 L2 权重正则化添加到电影评论分类模型中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "square-cleaners",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "30/30 [==============================] - 3s 60ms/step - loss: 0.5861 - accuracy: 0.7853 - val_loss: 0.4547 - val_accuracy: 0.8749\n",
      "Epoch 2/20\n",
      "30/30 [==============================] - 1s 29ms/step - loss: 0.3961 - accuracy: 0.8972 - val_loss: 0.4010 - val_accuracy: 0.8727\n",
      "Epoch 3/20\n",
      "30/30 [==============================] - 1s 34ms/step - loss: 0.3346 - accuracy: 0.9153 - val_loss: 0.3652 - val_accuracy: 0.8910\n",
      "Epoch 4/20\n",
      "30/30 [==============================] - 1s 33ms/step - loss: 0.3047 - accuracy: 0.9224 - val_loss: 0.3636 - val_accuracy: 0.8874\n",
      "Epoch 5/20\n",
      "30/30 [==============================] - 1s 31ms/step - loss: 0.2858 - accuracy: 0.9293 - val_loss: 0.3634 - val_accuracy: 0.8849\n",
      "Epoch 6/20\n",
      "30/30 [==============================] - 1s 29ms/step - loss: 0.2707 - accuracy: 0.9353 - val_loss: 0.3666 - val_accuracy: 0.8844\n",
      "Epoch 7/20\n",
      "30/30 [==============================] - 1s 30ms/step - loss: 0.2604 - accuracy: 0.9377 - val_loss: 0.3750 - val_accuracy: 0.8801\n",
      "Epoch 8/20\n",
      "30/30 [==============================] - 1s 30ms/step - loss: 0.2516 - accuracy: 0.9427 - val_loss: 0.3674 - val_accuracy: 0.8847\n",
      "Epoch 9/20\n",
      "30/30 [==============================] - 1s 34ms/step - loss: 0.2517 - accuracy: 0.9398 - val_loss: 0.3866 - val_accuracy: 0.8775\n",
      "Epoch 10/20\n",
      "30/30 [==============================] - 1s 29ms/step - loss: 0.2376 - accuracy: 0.9459 - val_loss: 0.4188 - val_accuracy: 0.8652\n",
      "Epoch 11/20\n",
      "30/30 [==============================] - 1s 35ms/step - loss: 0.2402 - accuracy: 0.9439 - val_loss: 0.3800 - val_accuracy: 0.8790\n",
      "Epoch 12/20\n",
      "30/30 [==============================] - 1s 36ms/step - loss: 0.2335 - accuracy: 0.9477 - val_loss: 0.3915 - val_accuracy: 0.8772\n",
      "Epoch 13/20\n",
      "30/30 [==============================] - 1s 34ms/step - loss: 0.2300 - accuracy: 0.9481 - val_loss: 0.4133 - val_accuracy: 0.8723\n",
      "Epoch 14/20\n",
      "30/30 [==============================] - 1s 38ms/step - loss: 0.2257 - accuracy: 0.9522 - val_loss: 0.3982 - val_accuracy: 0.8773\n",
      "Epoch 15/20\n",
      "30/30 [==============================] - 1s 38ms/step - loss: 0.2194 - accuracy: 0.9551 - val_loss: 0.4549 - val_accuracy: 0.8597\n",
      "Epoch 16/20\n",
      "30/30 [==============================] - 1s 36ms/step - loss: 0.2220 - accuracy: 0.9502 - val_loss: 0.3996 - val_accuracy: 0.8762\n",
      "Epoch 17/20\n",
      "30/30 [==============================] - 1s 42ms/step - loss: 0.2164 - accuracy: 0.9530 - val_loss: 0.4147 - val_accuracy: 0.8733\n",
      "Epoch 18/20\n",
      "30/30 [==============================] - 1s 33ms/step - loss: 0.2125 - accuracy: 0.9555 - val_loss: 0.4071 - val_accuracy: 0.8763\n",
      "Epoch 19/20\n",
      "30/30 [==============================] - 1s 34ms/step - loss: 0.2191 - accuracy: 0.9538 - val_loss: 0.4209 - val_accuracy: 0.8726\n",
      "Epoch 20/20\n",
      "30/30 [==============================] - 1s 39ms/step - loss: 0.2075 - accuracy: 0.9573 - val_loss: 0.5097 - val_accuracy: 0.8485\n",
      "time: 23.2 s (started: 2021-07-25 22:19:57 +08:00)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras import regularizers\n",
    "\n",
    "model = keras.Sequential([\n",
    "    layers.Dense(16, kernel_regularizer=regularizers.l2(0.002), activation=\"relu\"),\n",
    "    layers.Dense(16, kernel_regularizer=regularizers.l2(0.002), activation=\"relu\"),\n",
    "    layers.Dense(1, activation=\"sigmoid\")\n",
    "    ])\n",
    "\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "history_l2_reg = model.fit(train_data, \n",
    "                           train_labels,\n",
    "                           epochs=20, \n",
    "                           batch_size=512, \n",
    "                           validation_split=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "matched-chair",
   "metadata": {},
   "source": [
    "l2(0.002) 表示该层权重矩阵中的每个系数都会将 0.002 *weight_coefficient_value ** 2 添加到模型的总损失中。 请注意，由于此惩罚仅在训练时添加，因此此模型在训练时的损失将远高于测试时\n",
    "时间。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "accepting-supplier",
   "metadata": {},
   "source": [
    "图 5.19 显示了 L2 正则化惩罚的影响。 如您所见，具有 L2 正则化的模型比参考模型更能抵抗过度拟合，即使两个模型具有相同数量的参数。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "trying-quantum",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gstk6zidd0j31aa0ts41n.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "developing-behavior",
   "metadata": {},
   "source": [
    "作为 L2 正则化的替代方法，您可以使用以下 Keras 权重正则化器之一。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "public-police",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.regularizers.L1L2 at 0x7f88df3eebb0>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 3.67 ms (started: 2021-07-25 22:23:51 +08:00)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras import regularizers\n",
    "\n",
    "# L1正则化\n",
    "regularizers.l1(0.001)\n",
    "\n",
    "# L1+L2正则化\n",
    "regularizers.l1_l2(l1=0.001, l2=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afraid-venue",
   "metadata": {},
   "source": [
    "请注意，权重正则化更常用于较小的深度学习模型。 大型深度学习模型往往过于参数化，以至于对权重值施加约束对模型容量和泛化没有太大影响。 在这些情况下，首选不同的正则化技术：dropout。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rational-shepherd",
   "metadata": {},
   "source": [
    "**增加Dropout**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "improved-cuisine",
   "metadata": {},
   "source": [
    "Dropout 是最有效和最常用的神经网络正则化技术之一，由 Geoff Hinton 和他在多伦多大学的学生开发。 应用于层的 Dropout 包括在训练期间随机丢弃（设置为零）该层的许多输出特征。 假设一个给定的层通常会在训练期间为给定的输入样本返回一个向量 [0.2, 0.5, 1.3, 0.8, 1.1]。 应用 dropout 后，该向量将随机分布一些零条目：例如，[0, 0.5,\n",
    "1.3, 0, 1.1]。 丢失率是被归零的特征的分数； 它通常设置在 0.2 和 0.5 之间。 在测试时，没有单元被丢弃； 相反，该层的输出值按等于 dropout 率的因子缩小，以平衡比训练时更多的单元处于活动状态这一事实。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "antique-calvin",
   "metadata": {},
   "source": [
    "考虑一个包含层输出的 NumPy 矩阵，layer_output，形状为 (batch_size, features)。 在训练时，我们随机将矩阵中的一小部分值归零："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "periodic-sapphire",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 训练时\n",
    "layer_output *= np.random.randint(0, high=2, size=layer_output.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fabulous-artist",
   "metadata": {},
   "source": [
    "在测试时，我们按丢弃率缩小输出。 在这里，我们按 0.5 进行缩放（因为我们之前减少了一半的单位）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "light-television",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 测试时\n",
    "layer_output *= 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "underlying-alabama",
   "metadata": {},
   "source": [
    "请注意，此过程可以通过在训练时执行这两个操作并在测试时保持输出不变来实现，这通常是实践中的实现方式（见图 5.20）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "anonymous-hopkins",
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_output *= np.random.randint(0, high=2, size=layer_output.shape) \n",
    "layer_output /= 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "reliable-saver",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gstkekuqcaj31be0jq41h.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fitting-practice",
   "metadata": {},
   "source": [
    "这种技术可能看起来很奇怪和随意。 为什么这有助于减少过拟合？ Hinton 说，他的灵感来自银行使用的欺诈预防机制等。 用他自己的话说，“我去了我的银行。柜员一直在变化，我问其中一个为什么。他说他不知道，但他们经常搬家。我想这一定是因为这需要双方合作。 员工成功地欺骗了银行。这让我意识到，在每个样本中随机删除不同的神经元子集可以防止阴谋，从而减少过度拟合。” 核心思想是在层的输出值中引入噪声可以打破不重要的偶然模式（Hinton 称之为阴谋），如果不存在噪声，模型将开始记忆。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "christian-springer",
   "metadata": {},
   "source": [
    "在 Keras 中，您可以通过层在模型中引入 dropout，该层应用于它之前的层的 Dropout 输出。 让我们在 IMDB 模型中添加两层，看看 Dropout 在减少过拟合方面做得如何。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "frozen-latin",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "30/30 [==============================] - 3s 59ms/step - loss: 0.6379 - accuracy: 0.6217 - val_loss: 0.5094 - val_accuracy: 0.8311\n",
      "Epoch 2/20\n",
      "30/30 [==============================] - 1s 27ms/step - loss: 0.5200 - accuracy: 0.7561 - val_loss: 0.4224 - val_accuracy: 0.8668\n",
      "Epoch 3/20\n",
      "30/30 [==============================] - 1s 31ms/step - loss: 0.4433 - accuracy: 0.8172 - val_loss: 0.3512 - val_accuracy: 0.8815\n",
      "Epoch 4/20\n",
      "30/30 [==============================] - 1s 31ms/step - loss: 0.3751 - accuracy: 0.8582 - val_loss: 0.3037 - val_accuracy: 0.8874\n",
      "Epoch 5/20\n",
      "30/30 [==============================] - 1s 36ms/step - loss: 0.3216 - accuracy: 0.8835 - val_loss: 0.2846 - val_accuracy: 0.8900\n",
      "Epoch 6/20\n",
      "30/30 [==============================] - 1s 33ms/step - loss: 0.2835 - accuracy: 0.9035 - val_loss: 0.2813 - val_accuracy: 0.8879\n",
      "Epoch 7/20\n",
      "30/30 [==============================] - 1s 32ms/step - loss: 0.2557 - accuracy: 0.9167 - val_loss: 0.2793 - val_accuracy: 0.8927\n",
      "Epoch 8/20\n",
      "30/30 [==============================] - 1s 35ms/step - loss: 0.2276 - accuracy: 0.9259 - val_loss: 0.2857 - val_accuracy: 0.8922\n",
      "Epoch 9/20\n",
      "30/30 [==============================] - 1s 28ms/step - loss: 0.2006 - accuracy: 0.9361 - val_loss: 0.2952 - val_accuracy: 0.8876\n",
      "Epoch 10/20\n",
      "30/30 [==============================] - 1s 30ms/step - loss: 0.1832 - accuracy: 0.9427 - val_loss: 0.3005 - val_accuracy: 0.8903\n",
      "Epoch 11/20\n",
      "30/30 [==============================] - 1s 35ms/step - loss: 0.1678 - accuracy: 0.9490 - val_loss: 0.3244 - val_accuracy: 0.8886\n",
      "Epoch 12/20\n",
      "30/30 [==============================] - 1s 33ms/step - loss: 0.1515 - accuracy: 0.9565 - val_loss: 0.3392 - val_accuracy: 0.8905\n",
      "Epoch 13/20\n",
      "30/30 [==============================] - 1s 30ms/step - loss: 0.1353 - accuracy: 0.9601 - val_loss: 0.3852 - val_accuracy: 0.8860\n",
      "Epoch 14/20\n",
      "30/30 [==============================] - 1s 33ms/step - loss: 0.1267 - accuracy: 0.9635 - val_loss: 0.3904 - val_accuracy: 0.8881\n",
      "Epoch 15/20\n",
      "30/30 [==============================] - 1s 31ms/step - loss: 0.1156 - accuracy: 0.9663 - val_loss: 0.4242 - val_accuracy: 0.8858\n",
      "Epoch 16/20\n",
      "30/30 [==============================] - 1s 32ms/step - loss: 0.1101 - accuracy: 0.9674 - val_loss: 0.4375 - val_accuracy: 0.8825\n",
      "Epoch 17/20\n",
      "30/30 [==============================] - 1s 30ms/step - loss: 0.1005 - accuracy: 0.9706 - val_loss: 0.4579 - val_accuracy: 0.8851\n",
      "Epoch 18/20\n",
      "30/30 [==============================] - 1s 27ms/step - loss: 0.0987 - accuracy: 0.9712 - val_loss: 0.4923 - val_accuracy: 0.8855\n",
      "Epoch 19/20\n",
      "30/30 [==============================] - 1s 34ms/step - loss: 0.0896 - accuracy: 0.9748 - val_loss: 0.4985 - val_accuracy: 0.8846\n",
      "Epoch 20/20\n",
      "30/30 [==============================] - 1s 33ms/step - loss: 0.0858 - accuracy: 0.9762 - val_loss: 0.5835 - val_accuracy: 0.8794\n",
      "time: 21.8 s (started: 2021-07-25 22:32:00 +08:00)\n"
     ]
    }
   ],
   "source": [
    "model = keras.Sequential([\n",
    "    layers.Dense(16, activation=\"relu\"),\n",
    "    layers.Dropout(0.5),\n",
    "    layers.Dense(16, activation=\"relu\"),\n",
    "    layers.Dropout(0.5),\n",
    "    layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "\n",
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])\n",
    "\n",
    "history_dropout = model.fit(train_data, \n",
    "                            train_labels,\n",
    "                            epochs=20, \n",
    "                            batch_size=512, \n",
    "                            validation_split=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "third-perspective",
   "metadata": {},
   "source": [
    "图 5.21 显示了结果图。 这是对参考模型的明显改进——它似乎也比 L2 正则化工作得更好，因为达到的最低验证损失有所改善。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "failing-journey",
   "metadata": {},
   "source": [
    "![](https://tva1.sinaimg.cn/large/008i3skNgy1gstkiqvg3zj31680te77a.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "driving-description",
   "metadata": {},
   "source": [
    "回顾一下，这些是在神经网络中最大化泛化和防止过度拟合的最常见方法：\n",
    "* 获取更多的训练数据，或者更好的训练数据。\n",
    "* 开发更好的特征。\n",
    "* 减少模型的容量。\n",
    "* 添加权重正则化（对于较小的模型）。\n",
    "* 添加dropout。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tough-complaint",
   "metadata": {},
   "source": [
    "## 章节总结\n",
    "* 机器学习模型的目的是：准确地概括从未见过的输入。 这比看起来更难。\n",
    "* 深度神经网络通过学习一个可以在训练样本之间成功的参数模型来实现泛化——这样的模型可以说是对训练数据的“潜在流形”进行了插值学习。 这就是为什么深度学习模型只能理解与他们在训练期间看到的非常接近的输入。\n",
    "* 机器学习中的根本问题是优化和泛化之间的紧张关系：要获得泛化，你必须首先对训练数据实现良好的拟合，但提高模型对训练数据的拟合一段时间后不可避免地会开始损害泛化。 每一个深度学习最佳实践都涉及管理这种紧张局势。\n",
    "* 深度学习模型的泛化能力来自这样一个事实，即它们设法学习近似其数据的潜在流形，因此可以通过插值来理解新输入。\n",
    "* 在开发模型时，能够准确评估模型的泛化能力至关重要。您可以随意使用一系列评估方法，从简单的保留验证到 K 折交叉验证和带改组的迭代 K 折交叉验证。请记住始终为最终模型评估保留一个完全独立的测试集，因为可能已经发生了从验证数据到模型的信息泄漏。\n",
    "* 当您开始研究模型时，您的目标首先是实现一个具有一定泛化能力且可能会过拟合的模型。这样做的最佳实践包括调整学习率和批量大小、利用更好的架构先验、增加模型容量或简单地训练更长时间。\n",
    "* 当您的模型开始过度拟合时，您的目标将转向通过模型正则化来提高泛化能力。您可以减少模型的容量，添加 dropout 或权重正则化，并使用提前停止。自然，更大或更好的数据集始终是帮助模型泛化的首选方法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "greatest-register",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:tensorflow]",
   "language": "python",
   "name": "conda-env-tensorflow-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "toc-autonumbering": true,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
